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fill  Oct  Ufltoc  Some  Parting  Thoughts  on  Survivability 

UUGOL  llUlGO  Modeling  and  Simulation 


Mark  Twai  n  sai  d  that  Adam,  the  first  man,  was 
really  in  a  great  position  when  it  came  to 
public  speaking  (or  writing  in  a  newsletter), 
because  he  knew  that  when  he  said  something  really 
good  nobody  else  had  said  it  before  him.  So  it  is  with 
some  trepidation  that  I  put  fingers  to  keyboard  to  try 
and  say  somethi  ng  noteworthy  about  where  I  think  sur- 
vivability  models  and  simulations  (M&S)  need  to  go 
from  here  at  the  risk  of  repeating  what  somebody  else 
may  have  said  better  already.  But  after  a  33-year  career 
doing  modeling,  simulation,  and  weapons  systems 
analysis,  I  do  have  a  few  things  I  eft  to  say.  I'd  really  just 
like  to  talk  about  three  things— new  M&S,  M&S  credi¬ 
bility,  and  old  M&S. 

First,  I'd  like  to  say  to  the  de/elopers  of  new  M&S 
architectures  and  new  M  &S  ( I  i ke J  MASS  and  AJ EM )  that 
if  you  build  it,  they  probably  won't  come  Just  because 
you  can  gdt  it  to  work,  doesn't  mean  anybody  else  will 
use  it.  It  may  be  a  great  innovation,  but  there's  a  fairly 
long  transition  period  bdtween  building  a  new  M&S  tool 
and  getting  somebody  else  to  use  it  on  a  regular  basis  If 
you  wonder  why  almost  nobody  is  using  the  JMASS 
models  yet,  it's  partly  because  analysts  have  yet  to  be 
convi  need  that  its  i  n  thei  r  best  i  nterest  to  spend  theti  me 
and  money  required  to  switch  from  ESAMS,  TRAP  or 
RADGUNS.  The  same  is  true  of  AJ  EM— users  have  to  be 
shown  that  there  are  capabi  I  iti  es  i  n  AJ  EM  that  they  can't 
get  i  n  J  SEM ,  SH  AZAM ,  or  COVART.  And,  they  have  to  be 
convinced  that  AJEM  will  be  a  stable  useabletool  that 
works  fortheirapplication.  Convi  nangthem  isgoingto 
take  time  and  a  lot  of  effort.  So  that  means  that  the 
JTCG/AS  and  the  JTCG/ME  are  going  to  have  to  plan 
support  for  existing  tools  for  some  significant  period  of 
time  or  they  risk  dropping  support  for  the  tools  that 
everyone  is  using  before  there  are  viable  replacements 
Additionally,  they  need  to  plan  for  more  training  in  new 
M&S,  like  AJEM,  to  get  peopleto  accept  and  use  them. 

Secondly,  atsomepointDoD  isgoingto  come  to  it's 
senses  and  realize  that  while  everything  it  does  relies  on 
M&S,  almost  nobody  can  demonstrate  that  they  work 
correctly.  That's  not  to  saythatM&S  have  no  credibili¬ 
ty — it's  just  that  in  a  lot  of  cases  there  is  no  document¬ 
ed  evidence  showing  where  a  model  works  and  where 
it  does  not.  People  who  develop  M&S  go  to  a  lot  of 
effort  to  convince  themselves  that  their  model  works 
correctly,  but  they  almost  never  write  down  what  they 
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did  so  that  they  can  convince  somebody 
else!  And  the  pressure  of  producing  new 
software  on  schedule  and  within  budget 
almost  always  guarantees  that  documenta¬ 
tion,  verification,  and  validation  (V&V)  fall 
off  the  table.  So,  new  software  develop¬ 
ments  are  no  better  "V&Vd"  than  "legacy" 
tools.  TheJTCG/AScan  best  serve  thei  r  DoD 
and  industry  customers  by  putting  more 
fundi  ng  and  effort  i  nto  V&V  and  document¬ 
ing  the  survivability  M&S  being  used  by  the 
community.  The  EC  AT  is  a  good  example  of 
what  can  be  done  (although  it  needs  to  be 
expanded),  as  are  the  continuing  efforts  by 
JASA  to  provide  standard  Accreditation 
Support  Packages  (ASP)  for  all  SURVIAC 
models  and  simulations. 

And  lastly,  at  the  risk  of  sounding  like  a 
curmudgeon,  just  because  something  is  old 
doesn't  mean  its  not  any  good.  A  "legacy" 
tool  is  one  that  has  been  used  and  honed 
over  the  years  into  a  thing  of  beauty,  and  a 
joy  forever.  Well,  maybe  not  forever.  But  the 
survivability  M&S  that  are  in  SURVIAC  today 
have  been  improved,  verified,  validated,  and 
used  over  many  years  by  many  users  for  a 
wide  variety  of  applications  They're  not  per¬ 
fect,  but  they're  getting  the  job  donei  and 
they're  being  constantly  enhanced  and 
improved  by  many  people  So  don't  be  in  a 
hurry  to  replace  them  just  for  the  sake  of  get¬ 
ting  somethi  ng  new  or  someth  i  ng  that's  sup- 
posedly  easier  to  "maintain."  You  may  wish 
later  that  you  had  I  eft  well  enough  alone 

And  that's  about  it.  I've  enjoyed  working 
with  theJTCG/AS  over  the  last  15  years  or  so, 
and  I  hope  to  keep  seei  ng  everybody  i  n  a  new 
i ncamation  starti  ng  next  year. 


Since  Dave  is  retiring  in  January  2002, 
we  asked  him  to  give  us  his  parting 
thoughts  and  suggestions. 

-Editor 
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Personal  Thoughts  to  our  Colleagues 
in  the  M&S  Survivability  Community 


I'm  delighted  that  this  issue  of  Aircraft 
Survivability  has  been  de/oted  to  the  sig¬ 
nificant  issueof  modeling  and  simulation 
(M&S).  Very  few  topics  conjure  up  as  much 
emotion  and  interest  as  that  of  M&S— its  role 
mission,  funding  management,  realism,  con¬ 
figuration  control,  and  future 

From  the  very  begi  nni  ng  of  my  professi  onal 
career,  computers  and  simulations  have  been 
constant  companions  My  first  exposure  to 
them  was  in  the  Biophysics  Department  at 
IBM's  Thomas  Watson  Research  Center, 
Yorktown,  New  York  in  the  early  60's,  while 
still  in  undergraduate  school.  It  was  a  privilege 
to  work  with  the  professionals  there  on  what 
was  to  become  the  world's  first  computer- 
based  teaching  machine 
As  I  recall,  an  old  IBM  650  and  704  were 
somehow  hardwired  together  with  an  input 
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station.  This  input  station  consisted  of  asmall  8x8foot 
wi  ndowless  room,  equi  pped  with  a  seat  and  a  boxy  vi  n- 
tage  IBM  electric  typewriter,  which  could  take  com¬ 
mands  typed  in  by  the  "students"  trying  out  the 
"teachi  ng  machine"  It  was  exciti  ng  (and,  at  ti  mes,  exas¬ 
perating)  as  the  fledgling  software  was  written,  correct¬ 
ed,  improved  and  applied  to  help  teach  student 
volunteers  how  to  read  and  write  German,  take  court 
stenography,  and  perform  elementary  statistics 

M  y  early  exposure  to  the  practi  cal  appl  i  cati  on  of  M  &S 
quickly  showed  me  that  it  is,  indeed  a  two-way  street. 
Computers  and  the  models  that  make  them  serve  us  do 
some thi  ngs  wel  I ,  some thi  ngs  not  wel  I ,  and  other  thi  ngs 
not  at  all.  Computers  don't  get  tired,  bored  or  show 
other  signs  of  emotion,  althoucji  I  have  heard  peopletry 
to  attribute  emotions  to  them  on  occasion.  They  can 
retrieve  and  manipulate  data  extremely  well.  They  are 
also  unforgiving  Use  a  zero  instead  of  the  letter  "o,"  or 
use  a  comma  i  nstead  of  a  period  and  they  al  most  seem 
proud  to  tel  I  you  that  you've  made  a  mistake  (again).  In 
a  word,  they're  a  tool— not  the  end. 

Over  the  past  couple  of  decades,  I  have  watched  as 
the  Department  of  Defense  (DoD)  has  made  multiple 
attempts  to  manage  its  growing  M&S  investments  by 
promulgating  policies  regarding  the  development  and 
use  of  M&S.  I  n  1994,  a  major  step  in  this  direction  was 
the  formation  of  the  Defense  Modeling  and  Simulation 
Office(DMSO)  to  serve  as  theDoD  lead  in  assuring  that 
our  M&S  plans  practices  and  investments  are  well- 
founded  and  executed.  These  responsibilities  are  found 
in  the  DoD  5000  Directives,  Regulations  and 
Instructions  The  Department  has  also  published  a  vari¬ 
ety  of  other  documents  relating  to  M&S  including  the 
DoD  M&S  Master  Plan,  Simulation-Based  Acquisition,  and 
other  si  mi  lar  documents 

The  problem  today  is  that,  while  there  are  regula¬ 
tions  di  recti  ves  charters  and  gui  del  i  nes  addressi  ng  DoD 
M&S,  few,  if  any,  are  being  taken  seriously  or  put  into 
practice  This  fact,  again,  was  recently  made  abundantly 
dear  as  I  asked  the  over  200  attendees  of  the  M&S 
Conference  hosted  in  February  2001  in  Reno,  Nevada  by 
thej  TCG/AS  to  take  a  pi  ece  of  paper  and  write  four  sen- 
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tencessummarizingthese  regulations  Only  two  or  three 
would  even  venture  a  public  answer. 

After  al I  has  been  said  and  done  i  n  DoD  M&S,  much 
more  has  been  said  than  done  While  policy  and  proce 
dures  are  needed  and  have  been  promulgated,  I  see  little 
indication  that  they  have  been  much  more  than  a  wish, 
an  i  ntenti  on,  a  desi  re  but  certai  nly  not  a  real  ity  or  even 
a  moving  force 

Another  data  poi  nt  that  brou^it  this  i  nto  focus  even 
more  was  a  recent  M&S  survey  which  theDDOT&E/LFT 
office  completed  and  published  earlier  this  year.  We 
examined  M&S  in  support  of  acquisition  prog'ams  to  try 
to  assessthe  state-of-affai  rsfor  a  representative  cross-sec¬ 
tion  of  current  defense  acquisition  programs  (22  to  be 
exact).  We  were  able  to  brief  every  Service  Acquisition 
Executive  the  DDR&E,  and  the  USD(AT&L)  on  the 
results.  From  one  came  the  observation,  "SBA. 
Simulation  Based  Acqui  siti  on?  That's  just  a  slogan;  it's  a 
bumper  sticker."  There  were  other  similar  comments 
from  those  briefed  reflecting  the  unanimous  diagnosis 
that  a  I  ess  than  healthy  stateof  M  &S  management,  fund- 
ing  and  application  presently  exists  in  DoD. 

We  al  I  know  that  to  get  a  remedy,  we  must  fi  rst  admit 
that  we're  sick,  and  then  identify  what  that  sickness  is 
The  survey  and  other  testimonials  provide  ample  proof 
that  these  di  recti  ves  and  regulations  have  gone  unheed¬ 
ed  for  the  most  part  and  there's  much  work  to  be  done 
before  DoD  complies  with  its  own  M&S  directives  Ldt 
me  I  i st  a  few  candid  comments  made  at  the  Reno  M&S 
conference  worth  noting— 

•  OSD  is  such  a  fragmented  organization  that  you  can 
find  any  opinion  you  want,  maybe  you'll  even  find 
a  good  one 

•  Working  with  military  instructions  is  I  ike  building  a 
sauna  out  of  ice  cubes 

•  All  models  are  wrong,  some  are  useful. 

•  Everyone  wants  commonality  of  models,  but  the/ 
want  it  in  their  way. 

•There's  no  such  thing  as  validating  a  model. 
Validation  isjust  a fai led  attemptto  falsify  a  model. 
You  might  argue  with  a  couple  of  these  points  but  it's 
cl  ear  that  much  remains  to  be  done 

What  are  some  of  the  M&S  challenges  faang  the 
Pentagon  today?  Let  me  share  my  own  personal  feel  i  ngs 
on  the  subject— 

•  Implement  those  DoD  directives  relating  to  M&S 
already  on  the  books 

•  Organize  an  M&S  management  enterprise  (eg., 
consortium)  within  DoD  to  directly  support  the 
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prog-am  manager  acquisition  community 
across  the  board  with  DMSO  in  lead  and 
adequate  resources  to  do  the  job. 

•  Gain  early  substantial  funding  from  pro- 
gam  managers  to  invest  in  M&S  enter¬ 
prise  either  by  mandate  or  by  major 
revision  in  PM  i ncentive structure 

•  I  mpl  ement  a  "M  i  crosoft-wi  ndows"  type  of 
configuration  control  across  M&S  used  in 
support  of  DoD. 

•  Reconcile  the  current  dichotomy  in  mod¬ 
eling  supporting  the  training/ readiness 
community  versus  the  modeling  support¬ 
ing  the  materiel  acquisition  community 

•  Make  M&S  deliverables  a  requirement  for 
each  acquisition  program  to  the  maximum 
extent  possible 

•  Stop  looki  ng  at  M  &S  as  a  replacement  for 
realistic  testing.  T&E  and  M&S  are  part¬ 
ners,  not  competitors 

•  Stop  advertising  that  M&S  will  initially 
save  money.  For  a  period  of  ti  me  i  n  fact,  it 
will  actually  cost  more  to  address  M&S 
shortfalls  since  realistic  T&E  and  realistic 
M&S  must  both  be  done  to  address  cur¬ 
rent  W&A  issues 

•  Expand  application  of  M&S  to  also  assist 
in  experimental  design. 

•  Build  into  M&S  by  means  of  comment 
cards,  monitor  screen  warnings  and 
paper  output,  the  shortfalls  and  assump¬ 
tions  intrinsic  in  the  M&S  being  used  as 
well  as  information  on  the  configuration 
control  POC. 

H  avi  ng  made  the  above-menti  on  ed  propos¬ 
al  s  for  the  broader  M&S  community,  what 
challenges  face  the  survi vabi I ity/vul nerabi I i- 
ty/lethality/M&S  communities?  Let  me  home 
in  on  several— 

•  We  must  develop  and  applyan  M&Sarchi- 
tecture  in  cooperation  with  DMSO,  which 
permits  a  meaningful  and  workable  ana¬ 
lytical  bridgebetween  platform  state  (I e/el 
of  damage  and  mission  degradation)  and 
combat  effectiveness 

•  We  must  avoid  using  these  models  and 
simulations  as  promotional  gimmicks 

•  We  must  assure  that  the  most  widely  used 
models  supporting  the  Survivability  com- 

continued  on  page  9 
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Today's  complex  weapon  systems  and 
systems  of  systems  cannot  be  devel  oped 
and  evaluated  through  hardware-only 
test  and  evaluation  (T&E).  The  reasons  are 
many,  but  real  ly  come  down  to  a  cost  compar¬ 
ison  of  hardware  prototypes  and  hardware 
testing  in  the  appropriate  environments. 
Modeling  and  simulation  (M&S)  can  help 
reduce  those  costs  and  other  risks 
TheDoD  Simulation  Based  Acquisition  ini¬ 
tiative  promotes  the  use  of  simulations 
throughout  the  I ife  cycle  of  every  weapon  sys¬ 
tem,  and  among  all  weapon  system  de/el op- 
ment  prog^ams  There  are  tremendous  time 
and  dollar  savings  to  be  had,  and  perhaps  zero 
environmental  impacts  and  safety  worries  if 
we  can  wri  ng  out  systems  and  certify  thei  r  per¬ 
formance  using  M&S  before  we  e/e r  bend 
metal  in  production.  During  the  Army's  devel¬ 
opment  of  the  Crusader  cannon  artillery  sys¬ 
tem,  for  example  the  program  used  morethan 
150  simulations.  None  of  them  were  devel¬ 


oped  specifically  for  Crusader,  which  is  an  excellent 
example  of  how  simulation  reuse  can  save  time  and 
reduce  costs  Extensive  use  of  simulations  allowed  the 
Crusader  team  to  re-baseline  the  system  in  a  matter  of 
weeks  when  they  had  to  trim  morethan  40  percent  of  its 
weight  to  meet  the  Army's  future  I  ighter  force  requi re- 
ments 

However,  if  program  managers  can't  locate  the  M&S 
toolsthey  need,  and  if  after  they  locate  them  they  don't 
have  confi  dence  i  n  them  to  provi  de  "credi  bl  e"  ana/vers 
then  this  capability  is  of  no  value  The  challenge  for  the 
M&Scommunityinsu  pporti  ng  the  Department's  acqu  i  - 
sition  process  isto  make  M&S  resources  accessible  and, 
most  importantly,  credible 

To  appreci  ate  the  Department's  chang  ng  view  of  the 
importance  of  M&S  to  the  acquisition  process  take  a 
look  at  the  current  DoD  5000  series  of  acquisition  doc¬ 
uments  and  compare  them  to  what  they  replaced.  The 
use  of  the  words  modeling  and  simulation  have  gone 
from  a  mere  mention  in  a  subparagraph,  to  M&S  being 
emphasized  up  front  in  concept  exploration,  and 
planned  for  and  used  throucjiout  the  life  cyde  of 
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weapon  systems  Dr.  Delores  Etter,  former  Deputy 
Under  Secretary  of  Defense  for  Saence  and  Technology, 
saw  an  increasing  appetite  for  M&S  tools  as  advanced 
technologies  both  Government  and  commerdal,  prom¬ 
ise  significant  increases  in  M&S  capabilities  She  also 
noted  that  many  barriers  still  exist  in  M&S  (eg.,  human 
behavior  representation,  4-D  presentations)  and  sup¬ 
porting  technologes  (eg.,  software  intensive  systems 
processing  power  and  bandwidth),  but  emphasized  that 
DoD  must  continue  to  address  those  issues 

The  Defense  Modeling  and  Simulation  Office 
(DMSO),  as  the  Department's  focal  point  for  M&S,  has 
focused  its  attention  in  three  key  areas  that  are  benefit¬ 
ing  not  only  the  acquisition  community  but  the  train¬ 
ing  analysisand  experimentation  communities  as  well. 

The  first  area  is  architecture  The  DMSO-developed 
High  Level  Architecture (H LA)  isthe accepted  standard 
for  distributed  simulations  in  the  DoD.  Further,  it  is 
becoming  an  industry  and  international  standard  as 
well.  It  has  been  accepted  by  the  North  Atlantic  Treaty 
Organization  (NATO);  was  adopted  as  an  international 
standard  by  the  Object  Management  Group  (OMG)  in 
November  1998;  and  deared  the  final  hurdleto  becom¬ 
ing  an  Institute  of  Electrical  and  Electronic  Engneers 
(IEEE)  standard  in  September 2000. Thefunctionality of 
the  H  LA  is  based  on  the  premise  that  no  onesimulation 
can  satisfy  all  uses  and  users.  Itallows  an  individual  sim¬ 
ulation  or  set  of  si  mulations  developed  for  one  purpose 
to  be  used  for  another  purpose  by  federati  ng  them.  The 
collaborative  environments  established  at  the  Air  Force 
Research  Laboratory  at  Wri ght  Patterson  AFB,  Ohio,  and 
the  Army's  Night  Vision  Laboratory  at  Fort  Belvoir, 
Virgnia,  are  good  examples  of  using  the  HLA  in  an 
acquisition  process  Further,  the  HLA  complements  the 
Test  and  Training  Enabling  Architecture  (TENA)  in  pur- 
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pose  design,  development,  and  implementa¬ 
tion. 

The  second  area  of  DMSO  emphasis  ^envi¬ 
ronmental  representation.  DMSO  investments 
in  Integrated  Natural  Environment  (IN E)  proj¬ 
ects  have  lowered  barriers  to  the  use  of  M&S. 
INE  projects  include  the  Master 
Environmental  Library  (MEL),  a  one-stop 
shop  for  discovery  and  ordering  data  sources 
in  all  environmental  domains;  the  Synthetic 
Environment  Data  Representation  and 
InterchangeSpedfication  (SEDRIS)  that  allows 
standard  data  interchange  and  unambiguous 
data  representation;  and  the  Environment 
Scenario  Generator  (ESG)  that  offers  tailored 
scenario  generation  and  just-in-time-produc- 
tion  ordering.  What  are  the  benefits?  You  can 
build  tailored,  physically  consistent  scenarios 
to  achieve  test  plan  objectives  You  can  gener¬ 
ate  environmental  conditions  that  otherwise 
would  bedifficult  to  come  by,  such  asthosein 
denied  areas  of  the  world,  or  that  would 
require  risk  of  injury  or  damage  to  the  envi¬ 
ronment.  And,  you  can  operate  at  slower  or 
faster  than  real  time  to  focus  on  critical  system 
and/or  environmental  components 

Finally,  the  DMSO  focuses  on  its  role  in 
developing  DoD  M&S  policy,  espeaally  with 
respect  to  verification,  validation  and  accredi¬ 
tation  (W&A).  W&A  isthe  critical  process  for 
gaining  assurance  that  a  selected  model  or  sim¬ 
ulation  can  produce  "credible"  results  With 
community  involvement,  the  DMSO  led  the 
development  of  DoD  Instruction  (DoDI) 
5000.6L,  DoD  M&S  Verification,  Validation, 
and  Accreditation,  which  established  common 
terminology  and  defined  hicjvlevel  roles  and 
responsibilities  across  the  Department.  The 
DMSO  also  worked  with  theM&S community 
to  develop  the  Web-based  DoD 
Recommended  Practices  Guide  (RPG),  which 
defines  the  underlying  philosophy,  prinaples, 
and  methodologies  recommended  for  use  in 
DoD  W&A  efforts  You  can  review  the  RPG 
online  at  http://www.msiac.dmso.mil/wa/. 
While  Department-wide  application  of  those 
recommended  practices  will  help  assure  the 

continued  on  page  8 
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credibility  of  M&S,  each  of  the  Mi litary  servic¬ 
es  and  DoD  Components  are  responsible  for 
developing  W&A  policies  and  guidance  to  suit 
their  organizational  needs 

The  level  of  use  of  M&S  by  DoD  and  indus¬ 
try  is  a  paradigm  shift— and  the  paradigm  con- 
tinues  to  shift  as  technology  offers  us 
bigger-better-faster.  M&S  is  a  proven,  cost- 
effective  tool  for  getting  the  acquisition  job 
done  cheaper,  quicker,  safer  and  cleaner,  but 
taking  full  advantage  of  it  will  require  the  con¬ 
tinued  commitment,  cooperation,  education, 
and  evolution  of  the  M&S  and  acquisition 
communities 

The  DM  SO  can  promote  the  confident  use 
of  M&S  for  acquisition  through  the  establish¬ 
ment  of  standards  that  encourage  interoper- 
abi  I  ity  and  reuse  through  the  development  of 
environmental  representation  resources  that 
reduce  T&E  costs  and  risks;  and  throucfi  the 
promulgation  of  Departmental  policies  and 
practices  that  gui  de  the  development  of  credi¬ 
ble  models  and  simulations,  but  in  the  end 
assuring  the  credibility  of  M&S  for  Defense 
acquisition  and  T&E  is  a  task  that  has  to  be 
worked  at  all  levels 


Colonel  I/I/.  Forrest  Crain,  U.5.  Army,  has  served  as  the 
Director  of  the  Defense  Modeling  and  Simulation  Office 
since  March  1,  2000.  He  received  his  commission  from  the 
United  States  M ilitary  Academy  in  1975.  He  is  currently 
pursuing  a  doctorate  in  Information  Technology  at  George 
Mason  University.  He  has  served  in  a  variety  of  combat 
arms  positions  throughout  his  career.  Before  joining  the 
DM  SO  he  served  as  C hief  of  Strategic  Plans  for  the  M  ulti 
National  Division  (North)  in  Bosnia.  Colonel  Crain  turned 
the  directorship  of  the  DM  SO  over  to  Navy  Captain 
Michael  G.  Lilienthal  on  September  4  and  retired  from 
active  duty  on  September  7. 

Captain  Lilienthal  is  now  the  Director  of  the  Defense 
M  odelingand  Simulation  Office(DM  SO),  Alexandria,  Va. 
Captain  Lilienthal  is  a  triple  "Domer"  of  the  U niversity  of 
N  otre  D ame.  H e  received  a  B. A.  degree  in  Psychology  with 
a  minor  in  Physics  in  1973;  followed  by  a  M  .A.  degree  in 
Experimental  Psychology  in  1975;  and  a  Ph.D.  in 
Experimental  Psychology  in  1978.  He  was  commissioned  in 
1978.  Captain  Lilienthal  was  board  certified  as  a 
Professional  Ergonomist  (N o.  462)  in  1994.  Dr.  Lilienthal 
is  a  fellow  of  the  Aerospace  Human  Factors  Association  and 
the  Aerospace  M  edicai  Association.  H  e  is  also  a  member  of 
the  H  uman  Factors  and  Ergonomics  Society. 

Editor's  Note:  This  article  is  a  recap  of  Colonel  Crain's 
presentation  at  the  February  12-15  conference  on  “. Assuring 
Modeling  and  Simulation  (M&S)  Credibility  for  Defense 
Acquisition  and  T&E"  in  Reno,  Nevada. 


AJEM  Guru  Retires 


Tom  Wasmund  from  the  Naval  Surface 
Warfare  Center  at  Dahlgren,  Virgnia,  retired 
on  30  September  2001  after  over  40  years  of 
government  and  military  service  Tom  graduat¬ 
ed  from  the  University  of  Washington  with  a 
BSEE  in  1959,  and  started  his  career  as  a  Navy 
pilot  by  entering  flight  training  later  that  year. 
He  also  received  a  BSAE  and  a  MSAE  from  the 
Naval  Postgaduate School  in  1966  and  1967, 
respectively.  Among  his  other  contributions 
Tom  has  been  the  driving  force  behind  the 
development  of  the  Advanced  Joint 
Effectiveness  Model  (AJEM)  that  just  entered 
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service  this  year.  He  has  been  a  long  time  member  and 
chairman  of  various  vulnerability  and  lethal  ity  commit¬ 
tees  for  both  theJTCG/AS  and  theJTCG  for  Munitions 
Effectiveness  (JTCG/ME),  and  has  been  the  AJEM  Project 
Manager  since  1993.  AJEM,  sponsored  by  both  the 
JTCG/AS  and  JTCG/ME,  is  a  lethal ity/vul nerabi I i ty/end 
game  model  meant  to  account  for  al  I  types  of  warhead 
and  proj  ecti  I  e  ki  1 1  mechani  sm  effects  agai  nst  all  types  of 
ai  r  targets  i  n  a  si  ngle  model .  Tom  has  been  dted  by  the 
Chairmen  of  both  organizations  for  his  outstanding 
leadership  and  perseverance  in  successfully  carrying  this 
project  through  to  the  release  of  AJ  EM  1.0  i  n  July  of  this 
year.  We  wi  sh  Tom  the  very  best  i  n  hi  s  red  rement. 
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continued  from  page  5 

munity  are  promptly  W&A'd  down  to 
thei  r  source  codes,  correcti  ng  errors  where 
possible  and  publishing  cautions, 
assumptions  and  shortcomings  explicitly 
as  an  i  ntegral  part  of  the  M  &S  output. 

•  V\fe  must  migrate  to  best-of-breed  models 
and  cease  supporting  models  which  do 
not  reflect  the  state-of-the-art  for  a  given 
application. 

•  We  must  maintain  defective  and  current 
configuration  control  of  M&S  under  our 
purview. 

•  We  must  realistically  account  for  second¬ 
ary  damage  mechanisms  and  cascading 
damage 

•  We  must  migrate  from  empirically-based 
M&S  to  physics-based  vulnerabi  I  ity/lethal- 
ity  models 

•  We  must  fully  support  the  prog-am  man¬ 
agement  community  without  losing  our 
objectivity. 

•  We  must  educate  DoD  and  the  private  sec¬ 
tor  on  M&S  regulations,  requirements, 
capabilities  databases  and  methodologies 
avai  I  able  and  formal  i  ze  a  forum  to  ensure 
that  this  occurs  on  a  regular  basis 

•We  must  develop  meaningful  risk/cost- 
benefit  methodologies  which  can  support 
wise  investment  decisions  and  survivabili¬ 
ty  design  trades 

Survivability  and  lethality  M&S  and  testing 
continue  to  be  strongly  supported  by  the 
Administration.  The  following  response  to  a 
question  on  June  22,  2001  from  Senator  Strom 
Thurmond  by  the  Honorable  Michael  Wynne 
Prinapal  Deputy  Undersecretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  says 
it  well— 

“Live  Fire  Testing  is  an  important  and  integral 
part  of  the  Department's  weapon  system  test  and 
evaluation  process,  providing  timely  and  accurate 
assessments  of  system  survivability,  vulnerability, 
and  munitions  lethality.  Live  Fire  Testing  also  pro¬ 
vides  insights  into  methods  of  reducing  the  vulner¬ 
ability  and  improving  the  lethality  of  weapons  and 
weapon  platforms  assessing  battle  damage  repair 


capaomtiesana  issues  ana  improving  tne computer  moaenng 
of  weapons  system  lethality  and  vulnerability.  I  believe  strong¬ 
ly  that  Live  Fire  Testing  should  be  continued. " 

The  DoD  needs  to  redaim  its  leadership  in  M&S. 
Recall  that  the  very  first  computer  and  simulation  was 
built  for  the  U.S.  Army  under  a  contract  with  the 
University  of  Pennsylvania  for  the  purpose  of  produang 
ballistic  tables.  Furthermore  thefirst  image  generated  by 
a  computer  was  also  performed  by  the  Army,  that  of  a 
simplelinedrawing  of  an  XM-ltank,  some  30  years  ago 
at  the  Ballistic  Research  Laboratories  (now  ARL)  at 
Aberdeen  Proving  Ground.  While  the  private  sector 
community  has  transformed  the  world  of  computers  , 
the  ability  of  DoD  to  effectively  manage  its  M&S  has  not 
kept  pace  resulting  in  waste  duplication  of  effort,  and 
misuse  of  M&S  in  applications  for  which  th^  have  not 
been  validated. 

We've  come  a  long  way  from  the  simple  teaching  - 
mach  i  ne  of  the  earl  y  60's  andthesimplistic  two  d  i  men- 
sional  black  and  white  computer  drawing  of  an  Army 
vehide  and  for  that,  we're  all  grateful.  We've  gotten  to 
the  point  where  our  computer-generated  output  looks 
dose  enoucji  to  reality  that  we  could  mistake  it  for  the 
real  thing.  And  therein  lies  the  problem.  I  think  too 
many  in  our  community  have  extrapolated  this  leading- 
edge  computer-aided  design  (CAD)  capability  to  one  of 
assumi  ng  that  the  physics  of  vulnerability/lethality/sur¬ 
vivability  of  those  computer  representations  is  equally 
understood.  Nothing  could  be  further  from  the  truth. 
Colonel  Forrest  Crain,  DMSO's  Director  recently  stated 
that  "real  progress  in  M&S  will  be  determined  by 
demonstrated  benefits  not  policy  directives  or  rhdtoric" 

TheJTCG/AS,  working  in  close  cooperation  with  the 
Joint  Technical  Coordinating  Group  for  Munitions 
Effectiveness (JTCG/ME),  isuniquely  positioned  to  pro¬ 
vide  that  leadership  within  thesurvivability/vulnerabili- 
ty/ldthality  communities.  I  invite  all  readers  of  this 
publication  to  join  us  in  this  effort.  I  welcome  your  feed¬ 
back.  I  maybereachedatjobryon@doteosd.mil. 

Mr.  O'Bryon  serves  as  Deputy  Director,  Operational  Test 
and  Evaluation  in  the  Office  of  the  Secretary  of  Defense 
and  has  directed  the  Live  Fire  Test  Program  since  1986.  H  e 
also  has  oversight  responsibility  for  the  JTCG/AS  and 
JTCG/ME  programs  of  the  DoD.  He  is  a  graduate  of  The 
King's  College,  George  Washington  University  and 
M  assachusetts  Institute  of  Technology. 
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Modeling  and  Simulation  in 
Operational  Test  and  Evaluation 

by  Dr.  Marion  L.  W  illiamsand  Dr.  FrankGray 


Years  ago,  Clay  Thomas*  one  of  the  Air 
Force's  "wise  old  men,"  was  quoted  as 
saying  "Artifi dal  Intelligence  has  great 
potential— it  always  will  have"  Clay,  known 
for  hissubtlestatements,  would  probably  have 
said  the  same  thing  about  modeling  and  sim¬ 
ulation  ( M&S).  Over  the  past  few  years,  many 
expectations  for  modeling  and  simulation 
(M&S)  have  outpaced  capabilities  This  has 
certainly  been  the  case  with  respect  to 
Operational  Test  and  Evaluation  (OT&E) 
applications  Like  many  Department  of 
Defense  testers,  we  are  receiving  increased 
pressure  to  reduce  test  costs  by  substituting 
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M&S.  Certainly  M&S  can  increase  the  effiaency  of  the 
design  and  development  process  create  products  that 
work  better  after  fewer  prototypes  and  thus  save  time 
and  money  through  fewer  re-tests  Operational  testers 
should  beableto  take  advantage  of  these  same  simula¬ 
tions  However,  with  few  exceptions  modeling  and  the 
integ'ation  of  models  into  simulations  are  fragmented, 
underfunded,  and  pretty  much  the  exclusive  domain  of 
development  contractors  The  following  is  our  view  of 
how  M&S  should  fit  into  OT&E  and  what  we  think  is 
necessary  to  make  this  happen. 

First  the  Basics 

OT&E  is  defined  in  Title  10,  U.S.  Code  as  "the field 
test,  under  realistic  combat  conditions  of  any  item  of 
(or  key  component  of)  weapons  equipment,  or  muni¬ 
tions  for  the  purpose  of  determining  the  effectiveness 
and  suitability  of  the  weapons  equipment,  or  muni¬ 
tions  for  use  in  combat  by  typical  military  users  and 
the  evaluation  of  the  results  of  such  test."  Title  10  also 
says  that  OT&E  "does  not  include  an  operational 
assessment  based  exclusively  on  (a)  computer  model¬ 
ing  [or]  (b)  simulation."  None  of  this  precludes  appro¬ 
priate  uses  of  M&S  in  support  of  OT&E.  In  fact,  DoD 
5000.2R  stipulates  that  "test  planning  shall  consider 
M&S,"  and  "test  results  shall  also  be  used  to  da/el  op 
and  improve  model  sand  simulations."  It  further  states 
"when  adual  testing  is  not  possi  bleto  support  an  oper¬ 
ational  assessment,  such  assessments  may  utilize  com¬ 
puter  modeling  and/or  hardware  in  the  loop 
simulations  (preferably  with  real  operators  in  the 
loop)."  DOT&E  policy  also  permits  using  M&S  to  eval¬ 
uate  test  results  So  our  formal  direction  isto  use  M&S 
for  planning  better  tests  and  producing  bdter  evalua¬ 
tions  of  test  results  Not  to  replace  tests 

M&S  may  have  its  biggest  impad  in  planning  and 
predicting  operational  tests  During  planning,  simula¬ 
tions  can  hel  p  identify  more  profitable  areas  for  tesd  ng, 
define  data  and  instrumentation  requirements,  and  pro¬ 
vide  a  venue  for  running  fador-screening  experiments 
Test  predictions  can  significantly  improve  our  test 
process  by  maki  ng  sure  that  we  understand  what  to  I  ook 
for  and  to  help  us  recognize  when  events  depart  from 
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the  expected.  The  tri  ck  is  to  do  thi  s  at  a  cost  that  doesn't 
exceed  benefits 

Our  experience  has  been  that  simulations  needed  for 
test  planning  or  test  prediction  are  rarely  available  in  a 
form  usable  by  operational  testers  Usually  models 
developed  to  support  acquisition  are  internal  products 
of  the  developer.  The  models  are  not  documented,  are 
not  a  del  i  verable  and  the  results  of  specifi  c  studi  es  usi  ng 
the  model  s  are  not  i  n  a  fo  rm  rel  easabl  e  to  external  agen- 
cies  Wehave tried  to  develop  modelsforourown  OT&E 
use  or  to  significantly  modify  existing  models  But  the 
results  have  been  very  hi<^i  costs  ($10+ mi  I  lion  for  one 
progam)  and  minimal  benefits.  We  either  started  too 
late  or  ran  out  of  money—  or  both. 

On  a  more  positive  note  we  have  had  some  success 
in  using  models  for  operational  assessments  Air-to-air 
missile  de/elopers  routinely  use  modeling  for  system 
design  and  development,  and  operational  testers  have 
been  able  to  take  advantage  of  thi  s  i  nvestment.  A  recent 
example  is  the  AIM-9X,  where  the  program  office  and 
contractor  included  the  testers  from  the  begnning, 
incorporating  functionality  for  testers  and  faithfully 
induding  their  validation  requirements. 

Models  used  to  predict  test  results  have  been  more 
valuable  when  there  is  feedback  from  the  test  into  the 
model  to  correlate  the  results*  correct  model  inaccura¬ 
cies  or  modify  incorrect  assumptions  But  we  have 
learned  that  tests  must  be  planned  with  such  correlation 
in  mind.  It  has  been  very  difficult  to  correlate  models 
and  tests  after  the  fact.  In  one  particular  effort  designed 
to  correlate  model  and  test  results  we  were  never  sur¬ 
prised  when  the  model  of  the  test  item  was  used,  but 
were  always  surprised  when  the  actual  test  item  was 
used.  There  were  always  factors  i  n  the  test  that  were  not 
accounted  for  in  the  model.  Once  we  learned  of  the 
problem,  the  model  could  be  corrected. 

M&S  in  OT&E  will  become  cost-effective  tools  only 
when  we  learn  to  routinely  leverage  on  M&S  invest¬ 
ments  and  efforts  by  developers  There  needs  to  be  an 
integrated  M&S  plan  for  each  acquisition  program  out- 
lining  the  requirements  for  each  application,  and  with  a 
coordinated  modeling  effort.  Models  used  by  contrac¬ 
tors  for  design  should  be  madeavailablefor  other  appli¬ 
cations  including  OT&E.  There  needs  to  be  a  plan  for 
comparing  test  data  with  model  data  throughout  the 
acqui  siti  on  cyd  e  so  that  the  model -test-model  process  i  s 
continuous  from  initial  development  through  opera¬ 
tional  testing  and  employment.  In  many  instances  there 
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i  s  no  conveni  ent  way  for  thi  s  to  happen  except 
in  an  ad  hoc  manner. 

Use  of  models  in  direct  support  of  OT&E  is 
an  important  topic  but  largely  undefined.  We 
know  that  models  can  support  testing  but  our 
mission  is  to  put  the  real  system  in  the  real 
environment  with  real  operators  to  make  sure 
it  can  accomplish  its  operational  task.  We  fully 
understand  that  even  a  field  test  of  a  system  is 
a  simulation  of  the  true  operational  environ¬ 
ment,  but  it  provides  the  only  opportunity  to 
make  sure  that  the  system  works  not  only  as 
designed,  but  accomplishes  the  intended  mis¬ 
sion.  The  truest  source  of  OT&E  data  is  from  a 
real  test  item  in  a  test  environment.  Sometimes 
an  acceptable  substitute  is  the  real  test  item  in 
a  simulated  environment.  For  example  a  real¬ 
istic  electromagnetic  environment  is  often 
impossibleto  create  outdoors  So  weusefaal- 
itiessuch  as  anechoic  chambers  to  stimulatea 
real  test  item  with  a  dense  electronic  environ- 


continued  on  page  15 


11 


Air  Force  M&S  Policy 


A  dramatic  shift  has  occurred  in  the 
development  of  Air  Force  Modeling 
and  Simulation  (M&S)  policy.  This 
article  contrasts  the  old  and  new  ways  of 
developing  policy  and  addresses  Verification, 
Validation  and  Accreditation  (W&A)  as  a  spe¬ 
cific  topic. 

Headquarters  Air  Force,  Directorate  of 
Command  &  Control  (AF/XOC)  istheAir  Staff 
organization  responsible  for  developing  Air 
Force  M&S  policy.  This  policy  is  implemented 
through  directives  and  instructions  Secretary 
of  the  Air  Force  for  Acquisition,  Information 
Dominance  Directorate  (SAF/AQI)  is  the 
organization  on  the  staff  of  the  Secretary  of  the 
Air  Force  responsiblefor  M&S  policy  for  acqui- 


Figure  1.  Air  Force  M&S  Policy— Previous 
Structure 


sition.  Because  of  this  tie  the  two  organiza¬ 
tions  work  d  osel  y  together. 

M&S  Policy— The  Old  Way 

Figure  1  depicts  how  Air  Force  M&S  policy 
used  to  be  structured.  Air  Force  Policy  Directive 
(AFPD)  16-10  provided  general  guidance  and 
objectives  and  served  as  an  umbrella  for  Air 
Forcelnstructions(AFIs)  that  provided  specific 
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byLt  Col  Gerry  Smither  and  LtCol  Skip  Langbehn 

guidance  As  Figure  1  shows,  however,  the  AFIs  devel¬ 
oped  under  this  policy  did  not  have  a  consistent  theme 
or  focus.  Some  AFIs  were  M&S-centric,  such  as  those  on 
W&A,  Model  Management,  and  Data  Standards  while 
the  AFI  on  Acquisition  was  functionally  oriented. 
Extendi  ng  across  these  AFI  s  were  crosscutti  ng  functi  onal 
areas  such  as  acquisition,  test,  intelligence  and  logistics 
Each  AFI  was  first  drafted  within  AF/XOC  and  then 
coordinated  across  the  Air  Force  After  years  of  effort, 
only  theAFIs  on  W&A  and  Exercises&  Wargames  made 
it  through  the  coordination  cycle  successfully  and  were 
published. 

Unfortunately,  the  AFI  on  W&A  was  successfully 
coordinated  and  published  only  after  it  was  watered 
down  and  broadened  to  the  point  where  it  gave  only 
vague  guidance  making  it  practical  ly  useless  Why  didn't 
this  approach  to  AFI  s  work?  Before  answeri  ng  thi  s  ques¬ 
tion,  we  need  to  address  W&A  and  its  links  to  M&S  in 
the  decision  process. 

V&V — How  Much  Is  Enough? 

Models  and  simulations  provide  information  as  one 
of  many  in  puts  to  a  "ded  si  on-maker".  Thus  model  sand 
simulations  are  merely  tools  used  i  n  a  process  designed 
to  achieve  a  result.  Figure  2  outlines  this  dedsion-mak- 
ing  process  from  an  M&S  perspective 

An  informed  decision  is  made  by  the  decision¬ 
maker  based  on  alternatives  presented  by  the  trained 
analyst  who  has  done  a  cartful  analysis  grounded  in 
accredited  models  and  simulations  carefully  chosen 
from  a  strong  base  of  possibilities  that  have  been  ver¬ 
ified  and  validated  for  that  purpose  Drawing  condu- 
sions  (the  "deasion")  is  the  responsibility  of  the 
decision-maker.  The  purpose  of  the  analysis  is  to  pro¬ 
vide  critical  insists  and  present  a  cogent  summary  of 
results  to  the  decision-maker. 

From  an  M&S-centric  perspective  the  analysis 
depends  on  accredited  information  provided  by  a  suite 
of  verified  and  validated  models  and  simulations. 
Therefore  the  experienced  ded  si  on-maker  is  critically 
dependent  on  V&V.  In  reality,  the  decision-making 
process  rarely  works  as  shown  in  Figure  2. 

Figure  3  shows  what  really  happens  The  decision- 
maker  makes  an  informed  deasion— PERIOD! 
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Everythi  ng  el se  bel  ow  the  I  i  ne  may  or  may  not  i  nfl  uence 
the  decision-maker  and  thedeasion  made  The  tenuous 
chain  that  links  M&S  and  V&V  to  the  decision-maker  is 
easily  broken— with  good  reason. 

As  General  Shaud,  theformer  Chi  ef  of  Staff,  Supreme 
Headquarters  Allied  Powers  Europe  said,  "process  is  no 
substitute  for  careful  thought."  Results  from  M&S  are 
simply  one  of  many  inputs  to  the  dea  si  on-maker.  This 
begs  the  question  "how  much  V&V  of  the  models  and 
simulations  is  enoucji?"  The  M&S  party  line  says  that 
W&A  is  al  I  about  risk  reduction,  not  for  the  M&S  com¬ 
munity,  but  for  the  decision-maker.  Howe/er,  the  reality 


Figure  2.  Decision-Making 
Process — The  Theory 


when  V&V  is,  by  any  understanding  only  a 
small  part  of  the  story?  The  answer  is  that  it's 
easier  to  focus  on  the  "tools  and  toys"  rather 
than  on  the  people  and  processes.  To  com¬ 
pound  the  problem,  attempti  ng  to  write  a  sin¬ 
gle  policy  to  guide  V&V  policy  almost 
guarantees  a  "one  size  fits  al  I "  mi  ndsdt. 

As  Figure  4  illustrates,  a  "one-size"  V&V 
does  NOT  fit  al  I .  I  n  fad;,  there's  a  spectrum  of 
V&V  that  ranges  from  relatively  useful  to  prac¬ 
tically  useless.  One  line  of  the  diagram  indi¬ 
cates  the  relative  "usefulness"  of  V&V,  while 
the  other  line  i  ndicates  the  relative  situational 
compl  exi  ty.  The  I  eft  si  de  of  the  d  i  agram  i  s  char- 
acteri  zed  by  few  ^sterns  and  general  I  y  one  ver- 
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Figure  4.  The  Spectrum  of  V&V 


Figure  3.  Decision-Making  Process — The  Reality 

isthatdeasions  e/en  bad  ones,  will  be  made  regardless 
of  the  quality  or  quantity  of  V&V.  When  a  "bad"  dea- 
sion  happens  there  is  an  inevitable  search  forthe  guilty 
party.  V&V  won't  be  the  reason  for  the  bad  ded  si  on,  but 
overlooki  ng  it  makes  M&S  an  easy,  obvious  target.  Once 
M&S  isfound  guilty,  the  search  stops 
W&A  i  s  used  to  develop  a  clear  understandi  ng  of  the 
strengths  and  weaknesses  of  the  tools  being  used. 
Unfortunately,  all  the  V&V  in  the  world  will  not  stop  a 
bad  analysis  sidetrack  a  bad  deasion,  or  save  a  troubled 
program.  Given  all  of  this  why  do  we  continue  to  focus 
much  of  our  attention,  much  of  our  funding,  and  much 
of  our  effort  on  the  V&V  of  models  and  simulations 


sus  one  assessments  that  are  detai  led  but  show 
low  situational  complexity.  V&V  dominates  in 
this  real  m  and  tends  to  be  very  objective  and 
empirically  verifiable  Examples  can  be  found 
in  the  M&S  done  by  the  Test  and  Evaluation 
community  where  detailed  engineering  mod¬ 
eling  demands  rigorous  V&V. 

The  right  side  of  the  figure  represents  a 
chaotically  compl  ex  situation,  characterized  by 
thousands  of  systems  and  generally  force-on- 
force  assessments  that  are  empi  ri  cal  ly  unverifi- 
able  In  this  situation,  theories  abound  and 
facts  are  few.  Accreditation  dominates  in  this 
realm  and  V&V  tends  to  be  very  subjective 
Warfighting  deasionsfall  into  this  category— 

continued  on  page  14 
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as  the  situational  complexity  increases,  the  rel¬ 
ative  importance  of  V&V  decreases  The  bot¬ 
tom  line  is  that  V&V,  while  useful,  is  not  the 
key  player  in  command  decisions 

Perhaps  the  term  "W&A"  should  be  re¬ 
coined  as  "A  V&V  to  remind  us  of  the  spec¬ 
trum  of  V&V  that  exists  and  the  relative 
importance  of  Accreditation  and  Verification  & 
Validation.  As  author  Steven  Covey  reminds  us 
in  hisbook,  TheSeven  HabitsofH ighly Effective 
People,  "begin  with  the  end  in  mind." 
Althou^i  W&A  represents  the  chronological 
order  of  events  this  terminology  generally 
conveys  the  notion  that  Accreditation  is  an 
after-thought  and  that  a  "one  size  fits  all"  V&V 
exists  For  planning  purposes,  the 
Accreditation  deasion  and  process  must  be 
considered  first.  Thekeyto  successful  V&V  isa 
dear  understanding  of  the  decision  being 
mad$  the  decision-maker,  and  the  relative 
i  importance  of  the  i  nformati  on  bei  ng  provi  ded 


AF!  16-1002  j  AFI 16-1009  |  AF!  16-1004 


Figure  5.  Air  Force  M&S  Policy — New  Structure 

by  the  M&S.  Only  then  can  the  V&V  of  the 
models  and  simulations  be  properly  scoped 
and  planned. 

With  this  discussion  of  W&A  as  back¬ 
ground,  we're  back  to  our  original  question, 
"why  di  dn't  thi  s  approach  to  AFI  s  work?"  Even 
thoucji  the  current  AFI  on  W&A  was  success¬ 
fully  staffed  and  approved,  it  was  practically 
useless  because  the  guidance  it  provided  was 
too  broad.  In  a  briefing  to  (then)  Brigadier 


General  Sandstrom  of  AF/XOC,  one  author  of  thi  s  paper 
offered  the  following  choice  "we've  been  almost  spec¬ 
tacularly  unsuccessful  at  writing,  staffing,  and  getting 
coherent  'policy'  out  to  the  Air  Force"  Either  "the  avil- 
iansand  military... both  past  and  present,  are  unwilling, 
untrained,  unable  uneducated,  underachievers  too 
ignorant  to  properly  write  and  staff  a  package"  or  "we're 
going  about  this  wrong."  Fortunately,  Brigadier  General 
Sandstrom  chose  the  latter  reason  and  agreed  with  the 
condusion  that,  "we've  been  trying  to  write  broad  guid¬ 
ance  for  folks  who  need  specifics" 

Asa  result,  we  came  up  with  a  new  approach  to  devel¬ 
op  M&S  policy  by  using  Integrated  Product  Teams 
( I  PTs) .  By  rotati  ngFigurelbyninety  degrees,  the  focus 
is  now  on  the  functional  areas  and  how  M&S  should  be 
used  in  a  supporting  role  rather  than  writing  M&S  poli¬ 
cy  for  the  sake  of  M&S.  AFPD  16-10  now  acts  as  the 
umbrella  for  a  series  of  functionally  oriented  AFIs.  We 
formed  a  smal  I ,  ded  i  cated  I PT  wi  th  a  rotati  ng  co-chai  r  to 
develop  the  AFIs.  One  chairperson  was  drawn  from  the 
functional  lead  while  the  other  co-chair  is  from 
AF/XOC.  AFI  16-1002,  M&S  Support  to  Acquisition,  was 
the  first  AFI  developed.  Our  effort  was  chaired  by 
SAF/AQI  and  co-chaired  byAF/XOCA.  Asthefocus shifts 
to  a  different  functional  area,  a  different  IFT  member  is 
desi  giated  as  the  chai  rperson,  but  the  core  members  of 
the  team  remai  n  the  same  Thi  s  structure  al  I  ows  the  I  PT 
to  create  AFIs  that  are  linked,  address  a  specific  func¬ 
tional  problem,  and  incorporate  crosscutting  M&S 
issues  Finally,  having  the  same  IPT  members  has  the 
added  benefit  of  avoiding  "stovepiped"  approaches 
Without  a  "community"  or  functional  area  focus  we 
lose  support  for  M&S  Broader  issues  like  "W&A"  and 
"data  standards"  lose  meaning  and  support  when  sepa¬ 
rated  from  the  end  user.  Experience  shows  we  can  either 
write  ineffective  M&S  AFIs  that  attempt  to  be  all  things 
to  all  peopleor  write  tailored  AFIsthat  address  speafic 
functional  area  needs  This  AFI  team  is  designed  with 
cross-functional  representation  with,  as  a  minimum, 
representatives  from  logistics  test  &  evaluation,  acquisi¬ 
tion  and  intelligence  The  I  PT  drafts  each  AFI  by  identi¬ 
fying  problems  in  the  particular  functional  area  being 
addressed  that  the  use  of  M&S  could  help  resolve 
AFI  16-1002,  M&S  Support  to  Acquisition,  was  pub¬ 
lished  in  June  2000.  AFI  16-1003,  M&S  Support  to  Test 
and  Evaluation,  is  now  in  coordination  and  publication 
isplanned  for  the  Fall  timeframe  of  2001.  Thethird  AFI 
in  the  series  AFI  16-1004,  Threat  M&S  Validation,  is 
now  in  final  draft  form  and  ready  for  the  first  round  of 
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coordination.  This  functional  approach  allows  M&S 
issues  like  W&A  to  be  addressed  within  the  functional 
AFIs,  allowing  us  to  cancel  the  separate  AFI  on  W&A. 

Summary 

•  The  Air  Force  is  taking  a  functional  approach  to  the 
development  of  M&S  policy. 

•  Good  policy  helps  people  in  the  field  do  their  job 
better.  Don't  write  broad  guidance  for  folks  who 
need  specifics 

•  Modeling  and  simulation  isonlyasmall  partofthe 
story.  It's  important  to  understand  where  it  fits  in 
the  overall  "big  picture"  Don't  focus  on  the  tools 
and  toys  rather  than  on  the  people  and  processes 

•  W&A,  while  useful,  is  not  the  key  player  in  com¬ 
mand  decisions  Thereisno  "one  size  fits  all"  when 
it  comes  to  V&V,  but  rather  a  spectrum  of  V&V 
depending  on  the  situational  complexity. 

•  V&V  shoul  d  not  be  "the  tai  I  that  wags  the  dog"  I  Fs 
important  to  separate  the  "V&V"  from  the  "A."  A 
more  appropriate  term  for  "W&A"  is  "A  V&V"  to 
emphasize  that  Accreditation  should  determine 
both  the  scope  of  the  M&S  effort  and  the  V&V 
required. 

Lieutenant  Colonel  Gerry  Smither  was  a  co-chair  of  the 
Integrated  Product  Team  that  developed  AFI  16-1002,  M&S 
Support  to  Acquisition.  Until  recently,  Lt  Col  Smither  was 
the  Air  Force  Program  Element  M  onitor  for  M  odeling  and 
Simulation  working  in  the  office  of  the  Secretary  of  the  Air 
Force  (Acquisition),  Information  Dominance  Directorate, 
SAF/AQI.  This  office  is  responsible  for  acquisition  modeling 
and  simulation  policy  within  the  Air  Force  and  acquisition 
oversight  of  modeling  and  simulation  programs.  Lt  Col 
Smither  is  now  with  the  Single  Integrated  Air  Picture 
(SIAP)  System  Engineering  Task  Force 

Lieutenant  Colonel  Skip  Langbehn  was  a  co-chair  of  the 
Integrated  ProductTeam  that  developed  AFI  16-1002,  M&S 
Support  to  Acquisition.  Until  recently,  LtCol  Langbehn  was 
a  member  of  the  Air  Staff  as  Chief,  Analysis  Development 
Branch,  MS&A  Programs  Division,  Directorate  of 
Command  and  Control  (AF/XOC).  This  office  is  responsi¬ 
ble  for  Air  Force  M  odeling  and  Simulation  policy  and  pro¬ 
grams.  Lt  Col  Langbehn  is  now  with  the  Joint  Staff  in 
J -8/Forces  Division.  He  may  be  reached  at  skip. 
Iangbehn@  js.pentagon.mil. 


Modeling  and  Simulation  in 
Operational  Test  and  Evaluation 

continued  from  page  1 1 

ment.  But  we  don't  want  to  generate  opera¬ 
tional  test  results  with  a  model  of  the  test  item. 

Operational  testers  are  in  the  front  of  the 
pack  in  attemptingto  useM&Sto  improve  test¬ 
ing  Wewantto  do  itwhereitmakessense  and 
where  models  to  support  those  efforts  are 
already  available  We  are  not,  howe/er,  plan¬ 
ning  to  replace  operational  testing  with  simu¬ 
lation,  to  use  M&S  in  areas  where  it  doesn't 
make  sense  orto  use  inadequate  models 

**  * 
Dr.  Marion  L.  Williajns  is  the  chief  scientist, 
Headquarters  Air  Force  Operational  Test  and 
Evaluation  Center,  Kirkland  Air  Force  Base,  New 
M  exico.  The  center  is  responsible  for  testing  more 
than  250  major  programs  at  20  different  loca¬ 
tions  worldwide.*6erving  in  this  role  since  1980, 
he  provides  technical  direction  and  guidance  to 
theAFOTEC  commander,  the  headquarters  staff, 
the  AFOTEC  detachments  and  operating  loca¬ 
tions  and  various  external  organizations  to 
ensure  the  scientific  proficiency  of  Air  Force 
Operational  Test  and  Evaluation.  Dr.  Williams 
received  his  B.S.  in  Aeronautical  Engineering 
from  Texas A&M  University  in  1956,  hisM  .5.  in 
Mechanical  Engineering  from  the  University  of 
New  M  exico  in  1971,  and  his  Doctorate  Degree 
in  Industrial  Engineering  and  Management 
(Operations  Research),  from  Oklahoma  State 
University.  Dr.  Williams  is  a  Fellow  of  the  MORS 
and  recipient  of  the  prestigious  Vance  R.  W  anner 
Memorial  Award  for  contributions  to  military 
operations  research. 

Dr.  Frank  B.  Gray  is  the  Deputy  Technical 
Director,  Air  Force  Operational  Test  and 
Evaluation  Center,  Kirkland  Air  Force  Base,  New 
M exico.  He  is  Responsible  for  technical  oversight 
of  F-22,  Joint  Strike  Fighter,  and  other  tests  con¬ 
ducted  by  the  Special  Test  Directorate.  Dr.  Gray 
has  been  involved  with  test  and  evaluation  in  the 
Air  Force  since  1978.  He  received  his  B.S.  and 
M .S.  in  Aeronautical  Engineering  from  Ohio 
State  University,  and  his  Ph.D.  in  Industrial 
Engineering  and  Experimental  Statistics  from 
New  M exico  State U niversity. 
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David  H.  Hall 

by  Mr.  DaleB.  Atkinson 


Dave  H  al  I ,  who  has  been  a  mai  nstay  of 
theJTCG/AS,  has  decided  to  retire  in 
January  2002  and  we  want  to  take 
this  opportunity  to  recognize  Dave  for  his 
long  service  and  many  contributions  to  the 
JTCG/AS, JTCG/ME,  SURVIAC,  and  theoverall 
survivability  community  over  many  years 
Dave  grew  up  in  Lakewood,  California  and 
received  a  B.S.  and  M.A.  in  Mathematics  from 
California  State  University  in  1968  and  1972, 
respectively.  Dave  worked  hiswaythrough  col¬ 
lege  at  Disn^land,  first  selling  icecream  and 
then  picking  up  trash.  Dave  met  his  wife  Terri, 
his  freshman  year  in  golf  class  and  they  were 
married  in  June  1968. 

A  week  later,  Dave  went  to  work  in  the 
Fuze  Department  of  the  Naval  Weapons 
Center  that  was  originally  located  in  Corona, 
California.  He  worked  as  a  mathematician 
eval  uati  ng  fuze  concepts  for  Standard  M  i  ssi  I  e 
the  ZUNI  rocket,  a  non-nuclear  hard  point 
defense  mi  ssi  I  e,  and  other  rel  ated  systems.  I  n 
1972,  he  became  an  operations  research  ana¬ 
lyst  in  the  Weapons  Analysis  Division  where 
he  wrote  a  major  portion  of  the  Exploratory 
Development  Requirements  Document  on 
S&T  initiatives  for  Navy  airborne  weapons 
systems  and  conducted  analyses  to  support 
the  identified  requirements.  He  worked  on  a 
major  project  to  develop  and  implement  a 
methodology  to  evaluate  advanced  weapons 
system  design  concepts,  tactics,  and  force 
structure  that  involved  working  on  a  daily 
basis  with  operational  Navy  pilots.  He  also 
conducted  a  Lanchester  analysis  of  the  Yom 
Ki pper  War  that  i  ntroduced  hi  m  to  the  survi  v- 
ability  area. 

In  1977,  Dave  went  to  work  for  John 
Morrow  in  the  Survivability  and  Lethality 
Division,  and  has  worked  in  the  survivability 
and  lethality  area  ever  since  He  initially 
managed  a  project  to  develop  a  methodology 
for  evaluating  alternative  survivability 


enhancement  features  for  Marine  Corps  helicopters 
agai  nst  ai  r-to-ai  r  th  reats.  I  n  1979  he  became  head  of  the 
Ordnance  Systems  Evaluation  Branch  where  he  super¬ 
vised  ID  weapons  systems  analysts  performing  design 
analyses  of  anti-air  ordnanoe  for  a  variety  of  weapons 
systems  including  Phoenix,  Standard  Missile  Sparrow, 
AMRAAM,  AIAAM,  and  other  systems  in  exploratory 
development.  Dave  was  introduced  to  the  JTCG/AS 
when  he  attended  a  meeting  of  the  Vulnerability 
Analysis  Subgroup  at  Wright-Patterson  Air  Force  Base 
Dave  also  represented  the  Division  in  theJTCG/ME  as 
well  as  the  on  The  Technical  Cooperation  Program 
(TTCP)  fuze  and  analysis  working  groups  TTCP  is  a 
cooperative  effort  with  the  nations  of  Australia,  New 
Zealand,  Canada  and  the  United  Kingdom. 

In  1984  Dave  became  the  Chief  Analyst  of  the 
Survivability  Division,  providing  technical  overset  of 
all  division  projects  and  isalso  the  Branch  Head  of  the 
two  analysis  Branches  in  theDivision,  supervising  up  to 
30  scientists,  engineers  and  analysts  The  Survivability 
Division  is  responsible  for  developing  and  demonstrat¬ 
ing  ai  r  veh  i  cl  e  survi  vabi  I  i  ty  tech  no  I  ogy,  an  d  tracki  ng  the 
maturation  of  that  technology  into  thefleet.  Theanaly- 
sis  branches  are  responsi  blefor  eval  uati  ng  requi  rements 
and  technologies  in  the  disciplines  of  air  combat  sur¬ 
vivability  and  anti-air  missile  lethality.  Weapons 
Systems  supported  by  Dave  and  his  people  indude  the 
F-14,  F/A-18,  V-22,  A- 12,  AX,  A-6,  AV-8B,  H-60,  H-l,  H- 
46,  H-53,  Tomahawk,  as  well  as  others 

Si  nee  1987,  Dave  has  served  as  the  Chai  rman  of  the 
JTCG/AS  M  ethodol  ogy  Subgroup  ( now  the  Survi  vabi  I  ity 
Assessment  Subgroup)  coordinating  the  development 
of  survivabi  lity  methodology  across  the  Services  Dave's 
outstanding  methodology  expertise  and  friendly  and 
cooperative  manner  served  him  and  theJTCG/AS  well 
over  the  years.  The  Methodology  Subgroup  has  done  a 
lot  with  relatively  littlefunding,  under  rapidly  changing 
Servi  ce  and  0  SD  M  &S  i  n  iti  ati  ves,  and  changi  ng  ti  mes  i  n 
the  acquisition  world  in  general. 

From  1992  to  1996,  he  also  served  as  the  Joint 
Project  Manager  for  the  OSD  sponsored  Susceptibility 
Model  Assessment  and  Range  Test  (SMART)  project. 
SMART  developed  and  demonstrated  joint  model  and 
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simulation  verification,  validation,  accreditation,  and 
configuration  management  processes  now  being  used 
by  thejoint  Strike  Fighter  (JSF)  and  many  other  organ¬ 
izations  and  progams  This  effort  resulted  in  the  estab¬ 
lishment  of  thejoint  Accreditation  Support  Activity 
(JASA),  which  provides  M&S  support  services  to  many 
DoD  system  acquisition  programs,  such  as  the  JSF,  the 
F/A-18E/F,  AIM-9X,  ESSM,  RAM,  and  others  In  1997  he 
received  the  National  Defense  Industrial  Association's 
Survivability  Leadership  Award  for  planning  and  exe¬ 
cuting  the  SMART  project  that  da/eloped  "a  rational 
cost-effective  process  for  verifying,  validating,  and 
accrediting  (W&A)  models  that  support  acquisition 
deasions"  Dave  said  that  the  SMART  project  was  the 
most  interesting  major  project  he  was  involved  with 
over  the  duration  of  his  career,  and  he  really  appreciat¬ 
ed  the  opportunity  to  work  with  an  outstanding  team 
of  professionals*  from  both  Government  and  industry. 
Dave  also  said  that  the J  ASA  team  that  developed  from 
theSMART  project  has  been  a  joy  to  work  with  and  has 
provided  inval  uablesupport  to  anumber  of  acquisition 
progams 

Duringthistime,  Dave  also  represented  thell.S.  on 
the  TTCP  Panel  for  Weapons  Evaluation  where  he 
served  as  the  U.S.  Mission  Effectiveness  Focus  Officer 
and  led  a  key  technical  area  investigating  cooperative 
analysis  techniques  for  the  evaluation  of  advanced 
weapons  systems  technology  initiatives.  Dave  has  also 
been  the  Navy  JTCG/AS  member  of  the  SURVIAC 
Technical  Coordinating  Group  (TCG),  composed  of 
both  JTCG/AS  and  JTCG/ME  members.  The  TCG  pro¬ 
vides  technical  oversight  of  SURVIAC. 

During  1998  and  1999,  Dave  served  as  the 
Modeling  and  Analysis  lead  for  the  JSF  Survivability 
Integrated  Product  Team  (IPT)  under  the  JSF  Joint 
Program  Office's  Systems  Engineering  Directorate.  He 
was  responsi  bl  efor  coordi  nati  ng  survi  vabi  I  ity  method¬ 
ology  and  analysis  used  in  support  of  system  design 
and  evaluating  alternate  technology  options  Dave  is 
also  the  chairman  of  the  Naval  Air  Warfare  Center 
Weapons  Division  (NAWCWD)  Analysis  Resources 
Science  and  Technology  (S&T)  Network.  The  Network 
coordinates  analytical  resources  and  technologies 


across  the  various  sites  of  the  Center  and 
administrates  discretionary  funds  for 
improvements  in  analytical  capabilities  for 
eval  uati  ng  S&T  i  n  i  ti  ati  ves 

Dave  is  also  as  a  member  of  the  NDIA 
Combat  Survivability  Division  (CSD) 
Executive  Board  as  well  as  the  Chairman  of 
their  Strategic  Planning  Committee  Dave  has 
taught  the  analysis,  modeling,  and  simula- 
ti  on  part  of  Professor  Bal  I 's  survi  vabi  I  ity  short 
course  since  1985,  and  continues  to  do  so. 
Dave  has  received  numerous  awards  over  the 
years,  i  nd  udi  ng  bei  ng  sel ected  as  a  Fel  I ow  of 
the  Naval  Weapons  Center  in  1993  and  the 
Michel  son  Laboratory  Award  in  June  2001. 

Dave  is  very  proud  of  his  wife  and  family. 
His  wife,  Terri,  served  as  the  CEO  of  Desert 
Area  Resources  and  Training  (DART),  a  non¬ 
profit  agency  that  serves  devel  opmental  I  y  dis¬ 
abled  adults  and  children.  Terri  worked  there 
27  years  and  upon  leaving  recently,  DART 
announced  that  they  are  naming  a  new  build¬ 
ing  after  her.  FtADM  Johnson,  the  NAWCWD 
Commander,  also  presented  her  with  a 
plaque  for  her  service  to  the  Navy  and  the 
community.  Their  two  children,  Julie  and 
Jason,  are  both  grown  and  have  successful 
careers  of  thei  r  own.  Dave  and  Terri  moved  to 
the  coast  in  Carlsbad,  California  in  July  of 
this  year,  where  Terri  has  accepted  a  position 
as  Director  of  an  agency  serving  the  homeless 
Dave  is  commuting  to  China  Lake  until  he 
retires  and  then  plans  to  become  a  consultant 
upon  his  reti rement. 

On  behalf  of  theJTCG/AS  and  the  surviv¬ 
ability  community  asa  whole,  I  would  liketo 
acknowledge  Dave  as  someone  who  has 
made  a  difference  and  express  our  apprecia¬ 
tion  to  him  for  doing  such  an  outstanding 
job  over  the  years  We  wish  him  well  in  his 
new  career  as  a  consultant. 

M  r.  Dale B.  A tkinson  may  be  reached  at  dba@ 
starpower..net. 
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Survivability,  Lethality  and  Effectiveness — 

What’s  in  a  Word? 

by  Dr.  Paul  H.  Deitz 


Issues  of  survivability,  lethality  and  effec¬ 
tiveness  (S/L/E)  have  been  around  a  long 
time  In  fact  it  can  be  argued  that 
Operations  Research  (OR)  matured  as  a  disa- 
pline  during  World  War  II.1  In  afield  that  has 
been  subject  to  over  60  years  of  measurement, 
modeling,  and  analysis  one  might  assume  that 
assoaated  terms,  methods  and  phrases  should 
be  clear  and  pred se  to  al I  practiti oners  of  the 
art.  But  this  is  not  necessarily  the  case 
Surprised?  Well  then,  contemplate  the  follow¬ 
ing  questions— 

•  Why  do  all  S/L/E  metrics  seem  to  fall  in 
the  range  [0.0  <x  <1.0]? 

•  M ost  esti  mates  in  S/L/E  are  posed  i  n  terms 
of  probabilities  What  are  the  sources  of 
randomness? 

•Why  are  single  (S/L/E  event)  outcomes 
usually  described  as  probabilities? 

•  What  dimensions  if  any,  are  associated 
with  S^L/E  metrics? 


•  What  field  observables  are  comparable  to  the  stan¬ 
dard  S/L/E  model  metrics? 

During  the  onset  of  live-fire  testing  and  modeling  in 
the  1980s  the  answers  to  these  questions  weren't  so 
obvious  to  the  ballistic  vulnerability  workers  in  the 
Army.  The  i  niti  al  live-fi  re  ( LF)  tests  agai  nst  the  M 113  and 
then  the  Bradley  resulted  in  a  multitude  of  contentious 
issues  concerning  methodologies  for  choosing  muni- 
tion^hit  points  the  value  (if  any)  of  models  and  the 
comparability  of  modelsand  tests.  During  the  period  of 
the  Abrams  I  ive  fi  re  program,  some  new  strategies  were 
developed.2  At  the  foundation  of  those  strategies  has 
been  an  evolving  attempt  to  systematize  and  structure 
the  elements  of  5VL/E  in  order  to  darify  meanings  to 
establish  clear  causal  (i.e,  time-forward)  event  chains, 
and  illuminate  comparable  model-test  metrics35 
Figure  1  ill ustrates  one  such  effort  i n  that  process  I  n 
the  middle  of  the  figure  are  listed  six  specific  levels  or 
metric  d  asses  that  are  typically  encountered  in  an  end- 
to-end  weapon  analysis  Level  -1  embodies  the  charac¬ 
teristics  of  weapon  detection  and  identification.  Level  0] 
represents  theweapon  conditions  at 
warhead  launch.  L»/el  1  represents 
the  characteristics  of  a  weapon  at 
the  moment  it  begins  to  interact 
(eg.,  hit  or  detonate)  at  the  target. 
Level  2  represents  the  state  of  the 
target  components  (i.e,  killed,  not- 
killed)  after  interaction  with  the 
threat.  Level  3  represents  the  related 
platform  capability  states  (eg.,  abil¬ 
ity  to  move  communicate  sense 
engage  replenish)  of  the  platform. 
Level  4  represents  the  utility  state  of 
the  platform  (eg.,  ability  to  service 
usefullya  particular  mission). 
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Figure  1.  Levels  Relevant  to  Survivability,  Lethality,  and  Effectiveness.  Lethality  and  Vulnerability  are 
often  used  to  pursue  different  agendas.  Typically,  (Weapon)  Lethality  (left  side  of  diagram)  is  analyzed  begin¬ 
ning  with  the  weapons  launch  at  Level  0.  Thus  warhead  hit  dispersion  at  the  target  is  factored  in.  To  esti¬ 
mate  complete  weapons  Effectiveness,  weapon  target  detection  and  identification  must  be  included  as  well. 
By  contrast,  Vulnerability  (right  side  of  diagram)  normally  starts  with  an  assumed  hit  at  Level  1. 
Susceptibility  covers  all  of  the  prior  factors  leading  up  to  a  hit.  Survivability  provides  the  overall  accounting. 
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From  theTargdt  (or  Platform)  Orientation  shown  on 
the  ri  ght  of  Fi gu re  1,  the  i  ssue  of  Vul  n erabi  I  i ty  assu mes 
the  initiation  of  warhead-target  interaction  (Level  1)  and 
generally  proceeds  through  an  estimate  of  target  (or 
platform)  utility.  The  events  leading  up  to  warhead 
interaction  with  a  target  can  be  wrapped  in  the  term 
Susceptibility.  The  processes  of  Susceptibility  and 
Vulnerability  when  combined  end-to-end  can  be 
descri  bed  as  Survi vabi  I  ity. 

By  contrast,  from  the  Threat-Weapon  Orientation 
shown  on  the  I  eft  of  Figure  1,  Lethal  ity  indudes  not  only 
issues  of  target  engagement  (Level  1)  through  target  util¬ 
ity  (Level  4),  but  the  warhead  delivery  as  well;  hence  the 
addition  of  Level  0.  For  an  overall  estimate  of 
Effectiveness  many  analyses  also  indude  weapon  detec¬ 
tion-identification;  hence  the  addition  of  Level  -1. 
These  le/els  describe  a  complex  set  of  events  as  a 
sequence  of  physical  states  that  occurjust  as  one  physi¬ 
cal  or  engneering  process  ends  and  another  begins 
Thus  the  output  of  one  process  forms  the  i  nput  for  the 
next. 

To  complete  thi  s  paradigm,  we  requi  re  a  set  of  opera¬ 
tors  to  connect  the  physical  states.  In  Figure  2,  we  repeat 
the  description  of  the  Levels  or  physical -engineering 
states  on  the  left-hand  side  I  n  the  center  of  the  figure 
we  represent  each  Level  by  an  el  I  ipse  populated  by  vari¬ 
ous  state  vectors  (shown  as  bullets  •).  We  use  the  nota¬ 
tion  Onn+1  to  denotea  mapping  from  Le/el  n  to  Le/el 
n+1.  The  operator  abstraction,  On  n+1,  represents  a  way 
to  descri  be  actions  covered  in  one  of  the  foil  owing: 

•  Physics-based  tests  typical  ly  of  the  012  operator 

•  Engineering-based  tests,  typically  of  the02,3  oper¬ 
ator 

•  Operational  tests  that  specifically  focus  on  the034 
operator 

•  Developmental  tests,  that  examine  any  of  the 
mappi  ngs  between  Level  - 1  and  Level  3 

•  Mission  training  that  focuses  on  the  relationships 
between  Level  3  and  Level  4 

•  M&Sby  means  of  which  abstractions  are  de/el  oped 
to  represent  the  physical,  engineering,  or  opera¬ 
tional  activities  represented  by  each  of  the 
Operators 

It  is  this  last  property,  that  a  dual  is  formed  between 
actual  physical,  engineering  operational  states  and  test 
activities  (operators)  on  the  one  hand,  and  M&S  physi¬ 
cal,  engineering  operational  states  and  simulated  oper¬ 
ators  on  the  other,  that  holds  the  possi  bi  I  ity  for  model 
validation.  Clearly  validation  is  possible  only  if  the 
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model  abstractions,  as  described  by  the  Levels* 
and  the  physical  measurements*  gathered  via 
experimentation,  are  fully  equivalent.  Here  are 
a  few  observations  about  the  Levels— 

•However  many  I  e/els  one  chooses  to 
model,  the  mappings  are  generally  com¬ 
plex  and  stochastic 

•  The  metrics  of  Levels  -1  throucfi  3  are 
physically  based  and  measurable 

•  The  metrics  associated  with  Level  4, 
Effectiveness*  are  generally  not  measurable 

•  The  development  of  Probability  Density 
Functions  (PDFs)  for  each  metric  at  each 
Level  iscruaal 

•  The  results  at  a  given  level  are  due  to  all 
that  preceded  (i.e,  the  factorial  of  param- 
eter  growth  problem) 

•  There  are  no  state  vectors  at  any  Level 
described  by  a  probability.  Howe/er,  PDFs 
can  be  estimated  at  any  Level  by  comput¬ 
ing  multiple  state  vectors  and  observing 
fractional  occurrences  of  speafic  out¬ 
comes 

Given  this  level  of  problem  decomposition, 
it  is  easy  to  understand  the  challenges  LFT 
brou^it  to  the  M&S  domai  n.  Consider  a  vul- 
nerability  test  in  which  the  initial  conditions 
have  been  established.  Thissets  the  conditions 
at  Level  1,  right-hand  side  in  Figure  1.  This  is 
someti  mes  descri  bed  as  the  end-game  anal  ysi  s 
From  the  moment  when  a  warhead  strikes  a 
target  (or  a  detonation  occurs  in  proximity), 
the  following  operator  sequences  must  occur 
to  result  i  n  a  credi  ble  Level  4  uti  I  ity  esti  mate— 

0 1,2  Mapping:  Beginning  with  the  initial 
threat  mechanism,  eg.  fragment s),  jets,  blast, 
all  relevant  derivative  mechanisms  must  be 
inferred  as  well  (eg.  fire  smoke  shock).  These 
threats  must  then  be  juxtaposed  against  all  the 
target  components  with  which  they  can  inter¬ 
act.  Based  then  on  the  character  and  magni¬ 
tude  of  the  threats  and  the  susceptibility  of 
each  component,  a  decision  must  be  made  as 
to  whether  a  "kill"  has  occurred.  The  difficulty 
of  this  estimate  is  increased  when  multiple 
damage  mechani  sms  are  present  and  where  no 
si  ngl  e  threat  i  s  capabl  e  of  a  ki  1 1 ,  but  si  gnifi  cant 
damage  aggregation  can  occur,  none-the-less. 
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02,3  Mapping:  In  thisstep,  thecomponents 
judged  to  be  killed  by  the  01,2  Mapping 
process  along  with  those  properly  performing 
are  transformed  to  a  series  of  platform  per¬ 
formance  metrics  Examples  might  include 
main  armament  rate-of-fi re  number  of  G's 
turning  rate  the  number  of  feet  per  minute 
rate-of-climb.  The  process  can  be  performed  a 
number  of  ways;  in  some  cases  engineering 
models  exist  for  this  purpose  Often  fault  trees 
are  used  for  this  estimation. 

03i4  Mapping:  This  operator  takes  measur¬ 
able  performance  at  Level  3  and  transforms 
Measures-of-Performance  (MoPs)  to 
Measures-of-Effectiveness  (MoEs)  at  La/d  4. 
This  transformation,  couched  in  mission  per¬ 
formance  may  be  the  most  difficult  to  frame 
completely  and  objectively.  Clearly  there  is  a 


compl  ex  i  nterpl  ay  among  the  possi  bl  e  combi  nati  ons  of 
MoPsand  particular  MoEs 

MostlegacyV/L  models  have  tended  to  blur  the  phys¬ 
ical  and  engineering  processes  that  begin  at  Level  1  and 
end  at  Level  4.  Theda/dopment  of  physical  and  engi¬ 
neering  abstractions  for  each  operator  represents  a  com¬ 
plex  challenge;  if  these  transformations  are  prematurely 
combi  ned,  the  chances  for  a  credibl e  W&A  process  are 
greatly  diminished.  The  collapsing  of  the  operators 
between  Levels  1  and  4  is  an  example  of  what  can  be 
called  lumped-parameter  modeling.  By  this  kind  of 
approximation,  physical  and  engineering  complexity  is 
oversimplified  and  there  is  not  adequate  or  accurate 
parameterization  of  theddtermining  variables  Thus,  key 
intermediate  observables  are  not  estimated  and  final 
observables  are  insuffi  dent  to  validate  model  results 
Is  all  of  this  simply  notional?  Actually,  no!  The  vul¬ 
nerability  structure  illustrated  in  Figure  2  was  used  to 
construct  the  direct-fire  vulnerability  model  SQuASH  in 
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Figure  2.  A  Mapping  Abstraction.  The  ellipses  in  the  middle  column  represent  mathematical  spaces.  The 
bullets  (•)  contained  within  the  spaces  represent  vectors.  The  arrows  represent  mapping  operators  that  take 
a  vector  at  one  Level  and  perform  a  mapping  to  a  vector  at  the  next  lower  Level.  On  the  left,  the  descriptors 
for  the  various  Levels  are  given;  on  the  right,  the  actions  of  the  various  operators.  Note  the  single  connect¬ 
ed  path  of  arrows  from  Level  -1  to  Level  4  represents  a  single  sample  in  an  end-to-end  connected  process. 
As  shown  here,  the  initial  conditions  at  Level  -1  could  be  drawn  stochastically  or  input  deterministically. 
Since  many  of  the  succeeding  operators  (O1i0  to  03  4)  are  stochastic,  the  results  from  Level  0  through  Level 
4  usually  represent  a  set  of  connected  stochastic  metrics  in  outcome  space. 
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support  of  the  Army  Abrams  Live-Fi  re  program.  Later  the 
framework  was  used  to  generate  a  new  vulnerability 
envi  ronment  appropri  ate  to  di  rect-fi  re  and  i  ndi  rect-fi  re 
ground  and  anti-ai  r  targets6  It  was  also  used  to  i  improve 
greatly  the  context  for  the  estimation  of  personnel  casu¬ 
alties/  including  a  context  for  operational  effectiveness 
These  ideas  emerged  as  well  in  indirect-fire  code  to  sup¬ 
port  vul  nerabi  I  ityesti  mates  agai  nst  I  ogi  sd  cal  and  tacti  cal 
targets8-10  And  thissamecodeformed  thefoundation  for 
the  V/L  portion  of  the  Advanced  Joint  Effectiveness 
Model  (AJEM).M2 

Finally,  in  pursuit  of  the  important  DoD  mandate  to 
perform  code  W&A,  the  model-measure  dual  described 
above  has  contributed  to  improved  methodsto  compare 
intermediate  and  final  vulnerability  metrics13-15  One  of 
the  most  important  strategies  to  emerge  has  been  the 
practice  of  developing  probability  density  functions 
( PDFs)  for  each  class  of  model  metric  and  then  compar¬ 
ing  particular  test-derived  measurements  agai  nst  thedis- 
tributions  Due  to  sample  restrictions  single  outcomes 
may  serve  only  to  reject  model  algorithms;  however  as 
sample  sizes  increase  confidence  in  model  predictions 
can  be  built. 

Earl  ier  we  asked,  "What's  i  n  a  Word?"  Over  the  years 
some  of  the  words  used  by  the  community  to  represent 
various  metri  cs  and  processes  have  not  necessari  ly  been 
applied  in  the  same  way.  Theconstructsof  Figures  1  and 
2  were  developed  to  provide  better  predsion  and  more 
informed  meaning.  Armed  with  these  observations  it  is 
possibleto  build  more  rigorous  abstractions,  executable 
codes  and  credible  W&A  Strategies 

Dr.  Paul  H .  Deitz  is  the  Technical  Director  for  the  US. 
Army  Materiel  Systems  Analysis  Activity.  In  1964  he 
received  his  B.A.  in  Physic  from  Gettysburg  College, 
Pennsylvania.  He  earned  his  M S.  in  Engineering  and  his 
Ph.D.  in  Electrical  Engineering  from  University  of 
Washington,  Seattle  in  1971  and  1973,  respectively.  He  is 
the  Recipient  of  the  first  annual  Arthur  L.  Stein  M  emorial 
Cup  of  Excellence  in  Live-FireTest  and  Evaluation,  present¬ 
ed  on  15  January  1997  by  the  Director,  Operational  Test& 
Evaluation,  Office  of  the  Secretary  of  Defense, "...  for  con¬ 
tributions  as  a  national  leader  in  the  modeling  and  simula¬ 
tion  of  ballistic  Live-Fire  phenomena." 
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A  Meed  for  Speed 


by  Mr.  Jon  5.  Ogg 


Has  Acquisition  Reform  been  a  success 
or  a  failure  over  the  past  decade?  I 
will  argue  it  has  been  both,  depend¬ 
ing  upon  one's  perspective  From  the  vantage 
point  of  the  acquisition  community  it  is 
viewed,  overall,  as  positive  Under  the  banner 
of  acquisition  reform  we  have  seen  some  sig¬ 
nificant  effiaencies  in  contracting,  both  in  the 
busi  ness  sense  of  the  word  as  wel  I  as  i  ndustri- 
al  base  and  work  force  We  have  made  progress 
in  moving  to  a  more  streamlined  business 
model—  adopting  and  adapting  best  practices 
from  the  commercial  sector.  Further,  we  have 
de/eloped  long  term  relationships  with  our 
industry  partners  both  in  acquiring  weapon 
systems  and  sustaining  them  while  moving 
from  an  oversight  to  insight  rolein  conducting 
our  business  DoD  is  no  longer  leading  in  the 
development  of  key  technology  areas,  especial¬ 
ly  electronics  but  rather  leveraging  off  of  the 
continuous  refreshment  of  commerdal  tech¬ 
nology  broucjit  about  by  the  boom  in  the 
rapid  and  seemingly  endless  market  thirst  for 
better,  faster,  cheaper.  In  addition,  the  focus  or 
infatuation  on  reducing  waste  has  driven  a 
national  "Lean"  movement  that  is  serving  to 
propel  industry,  and  to  a  lesser  extent 
Government  towards  improving  effiaencies 
and  productivity.  This  in  turn,  is  driving  fur¬ 
ther  consolidation  or  contraction  of  both  the 
Government  and  industry  industrial  base 
resulting  in  fewer,  but  more  capable  produc¬ 
tion  and  sustainment  faalities. 

Another  dimension  to  this  reform  move¬ 
ment  is  nearly  a  50  percent  reduction  in  the 
human  capital  involved  in  acquisition  over  the 
past  decade  The  military  specifications/stan¬ 
dards  (specs^stds)  crusade  of  the  early  90s 
brought  about  by  then  Defense  Secretary  Perry, 
prodai  med  that  Government  specs/ stds  would 
be  the  exception.  This  government  standards 
reduction  and  the  myriad  of  other 
Government  streamlining  initiatives  has  pro- 
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vided  increased  flexibility  across  acquisition,  albeit  not 
without  some  attendant  collateral  damage  for  programs 
that  have  attempted  to  push  the  reform  to  the  I  i  mi  t  Yet, 
in  the  aggegate  the  rdbrm  movement  has  had  a  net 
positive  impact  on  the  way  we  acquire  and  sustain 
weapon  systems  for  our  ultimate  customers— the 
Warfighters 

As  for  the  counterpoint,  place  yourself  in  the 
Warfighter's  seat.  Has  reform  placed  more  capable  (bet¬ 
ter)  systems  into  their  hands  quicker  (faster)  and  at  less 
cost  ( cheaper)  ?  To  the  contrary!  We  actual  ly  have  si  gnif- 
icantly  increased  the  time  and  cost  of  placing  new  sys¬ 
tems  into  their  inventories  in  addition  to  driving  up  the 
cost  of  mai  ntai  ni  ng  these  systems  once  fi  elded.  So,  from 
their  viewpoint,  acquisition  reform  does  not  appear  to 
have  yielded  the  return  on  investment  prodaimed  by 
those  charged  with  delivering.  In  fact,  from  a  "50,000 
foot  view"  it  appears  to  have  had  the  opposite  effect. 
This  point  was  amplified  by  General  "Speedy"  Martin, 
Commander,  U.S.  Air  Forces  in  Europe  in  hisaddressat 
this  year's  National  Aerospace  Systems  Technology 
Conference  where  he  stated  the  enemy  is  "time"  Time 
i  s  used  i  n  the  context  of  both  provi  di  ng  capabi  I  i  ty  to  the 
field  and  integrating  information  to  enable  near-real- 
time  targeting.  Our  partial  solution  to  the  former  is  to 
introduce  the  concept  of  spiral  development  or  evolu¬ 
tionary  acquisition.  In  short,  deliver  incremental  capa¬ 
bility  sooner.  The  concept  is  not  new  but  yet  another 
adaptation  from  the  commercial  sector,  PCs  being  a 
model.  Provide  a  rudimentary  capability  at  a  reasonable 
price  with  ample  room  for  expansion  and  add  capabi  I  i- 
tythrough  hardware  and  software  over  time  at  an  added 
cost.  TheJ  oi  nt  Stri  ke  Fi  ghter  i  s  an  exampl  e  of  thi  s  model 
applied  to  weapon  system  procurement  For  $22  billion 
(TY$)  and  7.5  years  the  Warfighter  gets  a  basic  stealthy 
air-mud  capability.  Another  3+ years  and  $3  billion 
(TY$),  you  get  the  system  the  Warfi colter  needs.  Is  this 
"better,  faster,  cheaper?"  I  submit  not. 

So,  what  is  the  acquisition  community's  mission  or 
role?  To  continue  our  tradition  of  providing  unparal¬ 
leled  technology  in  the  form  of  weapon  systems  to  our 
customer?— a  gven.  Focusing  on  the  future  our  pio¬ 
neering  (make  a  difference)  mission  is  to  take  the  lead 
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in  moving  the  acquisition  community  into  the  position 
of  being  the  supplier  of  choice  rather  than  default  This 
will  entail  a  new  look  at  how  we  provide  capability 
while  reduang  cyde  time  and  attendant  costs  Our 
major  thrust  is  to  bring  Modeling  and  Simulation 
(M&S)  into  the  21st  century  as  THE  enabling  tool  for 
transforming  a  physical  model  —  testcentricacquisition 
process  into  a  virtual  model -simulation  centric  business 
under  the  banner  of  Simulation  Based  Acquisition 
(SBA),  or  more  appropriately  Simulation  Enabled 
Acquisition  (SEA).  I  belies  we  are  on  the  verge  of  a  rev¬ 
olution  in  the  development  process  brought  about 
through  confluence  of  advancements  in  M&S  fidelity 
and  stakeholder  acceptability,  coupled  with  the  rising 
costs  and  inability  to  test  the  systems-of-systems  in  a 
representati  ve  envi  ron  ment  Combi  nethi  s  state  with  the 
emerging  revolution  in  air  warfare  with  theintroduction 
of  Uninhabited  Combat  Air  Vehides  (UCAVs),  where 
M&S  is  the  principal  tool  for  deriving  capability  based 
requirements,  designing,  developing  and  verifying  the 
system,  and  you  have  an  environment  ripe  for  change 
The  recent  success  enjoyed  by  the  Predator  in  flawlessly 
executing  the  Hellfire  missile  integration  is  a  shining 
example  of  the  power  of  M&S  in  cutting  an  otherwise 
12-18  month  dfort  down  to  three  And  it's  only  the 
beg nni  ng...a  changing  "SEA state"  with  a  need  for  speed 
in  delivering  capability  as  the  catalyst. 

What  i  s  i  n  the  future  for  the  Ai  r  Force  i  n  general  and 
the  acquisition  community  in  particular?  My  money  is 
on  an  acceleration  of  Unmanned  Air  Vehicles/ 
Uninhabited  Combat  Aerial  Vehides  (UAVs/UCAVs)  at 
the  expense  of  conventional  systems  more  Science  and 
Technology  investment  in  key  enabling  technologes 
that  complement  UCAVs  to  include  advancements  in 
Intelligence  Surveillance  and  Reconnaissance  (ISR), 
weapons,  and  the  next  giant  leap  —  hydrocarbon  fueled 
hypersonic  propulsion,  a  prerequisite  for  making  rou¬ 
tine  accessto  space  affordableand  providing  for  a  stand¬ 
off  near  real-time  targeting  capability  (Missilier 
concept).  Also,  I  seea  return  to  the  60V early  70s  where 
we  demonstrated  the  realm  of  the  possible  through 
fl  ight  of  demos  (X-) ,  a  "Back  to  the  Future"  sequel .  And 
who  is  best  suited  to  champion  these  movements? 
Those  who  have  the  focus,  attitude  and  perseverance  to 
makea  difference  Is  it  not  the  21st  century  where  dreams 
of  yesteryear  are  the  real  ities  of  tomorrow? 


Mr.  Jon  S.  Ogg  is  the  Director,  Engineering  and 
Technical  M  anagement  D irectorate,  Aeronautical 
Systems  Center,  Air  Force  Materiel  Command, 
Wright-Patterson  AFB.  He  received  his  B.S  in 
aeronautical  and  astronautical  engineering,  his 
M.S.  in  aeromechanical  engineering,  and  com¬ 
pleted  the  coursework  for  a  PhD  in  engineering 
mechanics  from  Ohio  State  University,  and 
another  M.S.  in  business  management  from 
M  assachusetts  Institute  of  Technology.  H e  is  rec¬ 
ognized  as  the  Air  Force's  leading  authority  on 
integrity  programs  for  propulsion  and  power  sys¬ 
tems.  H  e  has  spent  25  years  of  service  in  acquisi¬ 
tion  and  has  been  involved  in  every  phase  of  a 
system's  lifecycle  including  serving  in  the  capacity 
of  chief  engineer  for  the  F-22  program  for  nearly 
a  decade.  H  e  helped  pioneer  the  current  integrat¬ 
ed  product  process  development  and  product  team 
approach  in  place  today  across  acquisition.  He 
may  be  reached  at  jon.ogg@  wpafb.af.mil. 
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MAN  PADS  Analysis 
Methodology  Development 

by  Mr.  Alex  G.  Kurtz  and  Dr.  Ronald  L.  H inrichsen 


Man  Portable  Air  Defense  Systems 
(MAN PADS)  have  become  a  preva¬ 
lent  threat  to  both  military  and 
civilian  aircraft.  In  recent  conflicts,  it  has  been 
proven  that  aircraft  have  survived  MAN  PADS 
encounters  Some  MANPADS  missiles  also 
failed  to  detonateon  or  withi  n  the  aircraft.  The 
survivability/vulnerability  aircraft  analysis 
community  is  beginning  to  understand  the 
critical  issues  relating  to  the  impacts  of  MAN- 
PADS  missiles  with  aircraft,  however,  thecom- 
munity  still  needs  a  validated  set  of  analysis 
tools  to  handle  this  threat.  In  recent  years  a 
series  of  aggressive  multi-year  programs  have 
been  initiated  to  address  these  voids  These 
programs  have  incorporated  parallel  efforts 
that  integrate  first  principle  high-fidelity,  non¬ 
linear  structural  analysis  codes  test  data,  and 
analytical/empirical  penetration  equations  to 
advance  the  state-of-the-art  in  vulnerability 
analysis  techniques  and  understanding  of  air- 
craft-M AN  PADS  encounters  This  article  pres¬ 
ents  a  snapshot  of  the  MANPADS 
methodology  development  projects. 

The  main  objective  of  these  efforts  is  to 
advance  aircraft  vulnerability  assessment  and 
predictive  methodologies  for  missile  encoun¬ 
ters  Speafic  objectives  are  to— 

•  Apply  advanced  finite  el ementy finite  dif¬ 
ference  structural  analysis  codes  to  the 
body-on-body  penetration  problem  and 
analytically  predict  missile  veloaty,  mis¬ 
sile  position,  penetration  depth,  degrada- 
tion  of  aircraft  structure  and  missile 
kineticenergyasafunction  of  time  and 
•  Develop  algorithms  to  adapt  the  Fast  Air 
Target  Encounter  Penetration  (FATEPEN) 
codeforusein  predicti  ng  MAN  PADS  mis¬ 
sile  encou  nters  wi  th  ai  rcraft  structu  res 
Figure  1  illustrates  two  distinct,  mutually 
supporting  parallel  code  development  efforts 
The  tri-Service  Advanced  Joint  Effectiveness 
Model  (AJEM)  uses  the  FATEPEN  methodolo¬ 


gy  to  predict  target  penetration  and  damage  FATEPEN 
analysis  algorithms  are  comprised  of  analytical/empiri¬ 
cal  eng  neering  type  terminal  ballistic  penetration  equa¬ 
tions  designed  to  provide  very  fast  run  times  for 
producti  o  n-type  analysisrunsThefiniteel  ement  anal  y- 
sis  (FEA)  uses  high  fidelity  physics-based  structural 
analysis  algorithms  which  account  for  the  material  den¬ 
sities  and  non-linearities  as  well  as  failure  strengths 
and/or  strains  of  both  the  MANPADS  and  target.  An 
expliat  time  integration  scheme  is  used  to  solve  the 
equati  ons  of  moti  o n  of  th e  bod i  es  as  they  make  contact, 
interact,  fai I,  and  move 


Figure  1.  Interaction  of  MANPADS  FATEPEN  and  Finite 
Element  Analysis  (FEA)  Development  Methodology 


TheNaval  Surface  Warfare  Center,  Dahlgren,  Virginia, 
throucfi  its  contractor  Applied  Research  Associates  of 
Denver,  Colorado,  is  responsible  for  the  FATEPEN  mod- 
els'algorithms  The  FATEPEN  models  are  based  as  much 
as  possible  on  fundamental  principles  of  mechanics 
together  with  assumptions  regarding  the  primary  load¬ 
ing  and  response  mechanisms  as  determined  by  experi¬ 
ment  or  first  prinaple  code  calculations  Under  the 
current  effort,  new  algorithms  are  being  de/eloped  to 
adapt  the  existing  penetration  models  to  partial  impact 
geometries  and  to  add  the  capability  for  time  resolved, 
multi-body,  penetration  and  damage  analysis,  both  of 
which  are  needed  to  improve  body-on-body  impact 
damage  predictions 
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The  46th  Test  Wing  Wright-Patterson  AFB  Ohio, 
through  it's  contractor,  University  of  Illinois  (Ul)  at 
Urbana-Champaign,  is  responsible  for  the  body-on- 
body  finite  element  analysis  impact  studies  The/  have 
built/obtained  finite  element  and  computer  aided 
design  (CAD)  aircraft  models  and  fabricated  one  FEA 
MANPADS  missile  The  MANPADS  missile  finite  ele¬ 
ment  model  was  constructed  in  detail  and  is  comprised 
of  di  screte  secti  ons  of  an  actual  mi  ssi  I  e  ( seeker,  warhead, 
guidance  and  control,  and  motor).  The  MANPADS  mis¬ 
sile  model  also  contai  ns  ddtai  led  data  on  section  geome¬ 
tries,  exterior  dimensions,  joint  construction,  joint 
strength,  component  construction,  material  properties, 
mass  properties  and  rocket  motor  case  strength.  Figure 
2  presents  a  collage  of  finite  element  predictions  of 
MANPADS  impacts  on  various  targets  In  each  frame  of 
the  figure  the  Von  M  ises  stress  contours  resulti  ng  from 
a  MANPADS  impacting  the  target  are  shown. 

Testing  is  critical  to  credible  modeling  and  simula¬ 
tions  ( M  &S) .  J  oi  nt  Li  ve  Fi  re  (J  LF)  i  s  not  chartered  to  co  n- 


Figure  2.  Collage  of  Finite  Element  Predictions  of 
MANPADS  Impacts  on  Various  Targets 


duct  validation  and  verification  (V&V)  of  the  analysis 
codes;  however,  when  opportunities  were  presented,  the 
MANPADS  analysis  development  programs  have  aug¬ 
mented  JLF  tests  to  extract  very  specific  data.  This  took 
the  form  of  camera  placement/ speed,  additional  strain 
gauges,  additional  blast  gauges,  and  additional 
accelerometers  spedfically  placed  to  augment  recent  or 
future  analysis  Data  were  used  to  verify  both  missile 
breakup  and  ai  rcraft  damage  Another  way  the  prog'ams 
are  conducting  incremental  V&V,  is  to  run  pre-test  pre¬ 
dictions  for  future  MANPADS  tests.  Fol  lowi  ng  the  tests 
the  code  developers  and  test  engineers  meet  to  discuss 
test/analysis  results  anomalies  and  data  voids 


To  ensure  credible  MANPADS  modeling 
and  simulation  methodology  d»/elopment, 
46th  Test  Wing,  Ul,  N3A/C,  and  ARA  collabo¬ 
rate  together  to  ensure  that  the  simulations, 
multi  pie  tests  and  analysis  programs  are  com¬ 
pletely  integrated. 

Althoucfi  the  codes  lack  warhead  and  blast 
algorithms  at  project  completion,  the  vul  nera- 
bility  analyst  will  be  able  to  use  FEA  proce¬ 
dures  and  higher  fidelity  physics-based 
body-on-body  algorithms  to  perform  detailed 
pre-test  predi  ctions  conduct  post-test  analyses 
and  investigate  potential  vulnerability  reduc¬ 
tion  techniques  He'shewill  also  be  able  to  use 
updated  FATEPEN  engineering  algorithms  to 
acquire  initial  MANPADS  body-on-body  air¬ 
craft  vulnerability  estimates  do  vulnerability 
"production"  runs,  and  conduct  design  trade¬ 
off  studies 

Mr.  Alex  Kurtz  received  his  B.S.  in 
Aeronautical/Astronautical  Engineering  from 
Ohio  State  University.  He  is  a  research  and  test 
engineer  for  the  46th  Test  1/1/  ing,  1/1/ right- Patterson 
AFB,  Ohio.  He  has  been  an  Aircraft  Survivability 
Specialist  for  14  years,  working  on  various  air¬ 
craft  vulnerability  reduction  programs  to  include: 
AC-130U  Gunship,  F-16,  C-130J,  C-5,  C-17,  F- 
111,  and  F-22.  He  is  currently  the  Chairman  of 
the  Aircraft  &  Crew  Protection  Committee  for  the 
JTCG/AS.  He  may  be  reached  at  937.255.6302 
extension  250  or  alex.kurtz@  wpafb.af.mil. 

Dr.  Ronald  L.  Hinrichsen  received  his  B.S.  in 
Aeronautical  Engineering  from  the  University  of 
Arizona  and  his M  .S.  and  Ph.D.  in  Aeronautical 
Engineering  from  the  Air  Force  Institute  of 
Technology.  He  is  a  Senior  Research  Scientist 
employed  by  the  National  Center  for 
Supercomputing  Applications,  University  of 
Illinois  at  Urbana/Champaign  (NCSA/UIUC). 
H  e  has  over  25  years  of  experience  in  the  teach¬ 
ing,  development,  and  application  of  composite 
materials,  solid  mechanics,  structural  dynamics 
aeroacoustics,  and  computer  aided  design.  He 
may  be  reached  at  937.904.5137  or 
orhinricrl@  asc.hpc.mil. 
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M&S  Credibility  Workshop 

Assuring  M&S  Credibility  for  Defense  Acquisition 
and  T&E  Survivability,  Lethality  and  System  Effectiveness 
12-15  February  2001,  Reno,  Nevada 

by  Mr.  David  H.  Hall 


Survivability,  lethality  and  mission  effec¬ 
tiveness  Models  and  Simulations 
(M&S)  are  a  key  element  of  support  to 
acquisition  program  milestones  Heavy 
rel iance  on  these  M &S  has  resulted  i n  an  ever- 
increasing  emphasis  on  the  ability  to  demon¬ 
strate  thei  r  credi  bi  I  ity.  At  I  east  one  recent  study 
has  shown,  however,  that  DoD  and  Service 
M&S  initiatives  provide  for  neither  long-term 
support  of  these  critical  M&S,  nor  for  assess¬ 
ment  of  their  credibility.  The  Director, 
Operational  Test  and  Evaluation  (DOT&E) 
commissioned  a  study  in  1999,  conducted  by 
Hicks  &  Associates  Inc.  (Modeling  & 
Simulation  Survey  Briefing  dated  January 
2000),  on  M&S  use  in  acquisition.  The  study 
showed  that  while  an  acquisition  program 
manager  (PM)  must  invest  early  in  M&S  to 
make  the  best  use  of  those  M&S,  a  number  of 
factors  hamper  his  or  her  ability  to  fully  bene¬ 
fit  from  that  investment.  The  study  found 
that— 

•  The  Services  vary  in  their  approaches  to 
M&S  management 

•Industry  is  the  predominant 
developer/owner  of  the  M&S  used  for 
acquisition 

•  Thereisoften  uncertainty  about  the  "pedi- 
gree"  of  M&S  used 

•  The  typical  PM  does  not  view  DoD-wide 
M&S  investments  as  cost  -or  schedule  - 
effective 

Asa  result,  the  suitability  and  credi  bil  ity  of 
M&S  used  to  support  acquisition  programs 
may  be  unknown  or  assumed.  The  DOT&E 
study  recommended,  among  other  actions* 
thatOSD  should  direct  the  implementation  of 
a  process  to  identify  and  satisfy  M&S  require¬ 
ments  for  acquisition  programs,  and  review 
and  clarify  roles  and  functions  of  DoD  M&S 
organizations 
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To  further  explore  the  condusions  of  the  DOT&E 
study,  the  Modeling  and  Simulation  Workshop  (held  in 
February  2001  in  Reno,  Ne/ada)  identified  the  unique 
problems  faced  by  acquisition  progams  using  surviv¬ 
ability  and  lethality  M&S.  The  Workshop  also  explored 
potential  solutions  to  these  problems  One  major  con- 
dusion  recognized  the  need  for  better  integration  of 
acquisition  program  M&S  requirements  into  OSD  and 
Service  policies  guidance  and  initiatives 

The  Workshop  was  constructed  around  presentations 
and  panel  sessions  by  OSD  and  Service  decision-makers, 
acquisition  program  managers  and  T&E  professionals. 
These  induded  DoD,  Industry,  academia,  and  the 
International  acquisition  community.  The  program  was 
divided  into  three  sessions:  OSD  View  of  M&S  Use  in 
Acquisition;  Acquisition  Program  M&S  Challenges  and 
Perspectives;  and  Test  and  Evaluation  Community  M&S 
Challenges  and  Perspectives  Mr.  James  F.  O'Bryon 
(Deputy  Director,  Operational  Test  and  Evaluation/Live 
Fi  re  Tesd  ng,  Offi  ce  of  the  Secretary  of  Defense)  gave  the 
keynote  Address 

Briefings  in  the  second  session  included  representa¬ 
tives  of  Air,  Sea,  and  Land  and  international  acquisition 
programs  usi  ng  M  &S  to  support  thei  r  requi  rements  The 
third  session  presenters  represented  Service  operational 
testing  organizations  intelligence  organizations  and 
representatives  of  test  and  e/aluation  fad  lities 

Each  day's  session  ended  with  a  panel  (comprised  of 
the  day's  presenters)  discussion  focused  on  Workshop 
objectives  The  panels  addressed  both  open  mike  and 
structured  questions  On  Monday  afternoon,  the 
Workshop  was  preceded  by  the  JASA  tutorial  entitled 
"Essentials of  Simulation  Credibility." 

The  Workshop  was  specifi cal  ly  structured  to  develop 
a  consensus  and  ensure  that  Workshop  products  prop¬ 
erly  reflected  the  views  of  the  majority  of  the  partici¬ 
pants  A  summary  of  the  previous  day's  discussions, 
condusions  and  observations  was  presented  at  start  of 
each  day's  session.  Thediscussion  of  that  summary  each 
day  was  aimed  at  de/el  oping  consensus*  with  significant 
audience  parti  a  pati  on  (both  during  and  after  the  daily 
summaries).  Strong  minority  opinions  were  captured 
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and  noted.  These  daily  summaries  (induding  the  third 
day's  summary,  which  was  not  discussed  at  the 
Workshop)  were  sent  to  all  parti  a  pants  via  E-mail  after 
the  Workshop  for  review  and  comment.  The  numerous 
comments  and  change  requests  were  summarized  and 
incorporated,  with  strong  mi  nority  opinions  recorded  in 
the  final  summary.  This  process  guaranteed  that  the 
results  and  conclusions  of  the  Workshop  reflect  a  true 
consensus  of  opinion  from  the  200  plus  workshop  par- 
tiapants.  In  addition,  post-Workshop  comments  reflect 
some  comments  from  parti  a  pants'  management  that 
were  not  i  n  attendance 

Conclusions 

Survivability,  lethality  and  system  effectiveness  M&S 
arekey  elements  of  support  to  acquisition  programs  and 
their  demonstrated  credibility  is  essential  to  these  pro¬ 
grams  ButDoD  and  ServiceM&Sinitiativesdo  not  pro¬ 
vide  for  long-term  assurance  and  maintenance  of 
credibility  of  theseM&S  used  in  acquisition.  Acquisition 
programs  and  the  T&E  community  are  by  and  large 
required  to  forge  their  own  paths  to  credible  integration 
of  M&S.  The  Workshop  results,  independent  studies 
and  J ASA's  di  rect  experi  ence  worki  ng  for  acqui  si ti  on  pro¬ 
grams  support  these  conclusions 

M&S  and  W&A  policies  and  technical  guidance  have 
been  made  available  to  the  acquisition  community 
through  the  Service  M&S  Offices  and  DM  SO.  The  "bot¬ 
tom  line"  condusion  of  the  three-day  workshop  was 
that  meaningful  implementation  of  these  polides  is 
hampered  by  several  factors  that  combine  to  lessen  their 
potential  benefits— 

•Inconsistent  M&S  and  W&A  definitions  and 
requirements  across  Services 

•Inconsistent  standards  applied  to  M&S  software 
development 

•  Severely  constrained  protyam  resources 

•Policy  "flexibility"  that  permits  a  minimalist 
approach  to  W&A 

•  Lack  of  an  OSD-le/d  focal  point  for  M&S  credibili¬ 
ty  in  acquisition 

•  Defense  Acquisition  Board  (DAB)  process  that  does 
not  i nd ude  any  ra/iews  of  M&S  credi bi I ity 

•  Lack  of  any  impacts  for  not  following  policies  on 
M&S  credibility 

•  Confusion  over  responsibilities  for  M&S  credibility 
generated  by  Acquisition  Reform 
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In  fact,  the  combination  of  factors  listed 
above  makes  it  quite  possible  for  an  acquisi¬ 
tion  program  to  comply  with  individual 
Service  and  DoD  M&S  and  W&A  policies  to 
"accredit"  a  simulation,  and  still  not  have  a 
robustl  y  eval  uated,  objedti  vel  y  credi  bl  e  si  mul  a- 
tion. 

Put  another  way,  "accredited"  simulations 
are  not  necessarily  "credible:'  simulations 
M&S  and  W&A  policy  compliance  is  a  neces¬ 
sary,  but  not  sufficient,  condition  to  demon- 
stratethecredibi  I  ity  of  a  simulation  objectively 
and  robustly.  The  general  assumption  at  OSD 
that  M&S  credibility  has  been  assured  by  the 
time  the  program  reaches  a  DAB  revie/v  needs 
to  be  revisited. 

It  was  dear  from  the  Workshop  briefings 
and  panel  discussions  that  Service  M&S  Offices 
and  DM  SO  do  not  want  the  role  of  "policing" 
compliance  with  M&S  and  W&A  policy,  and 
they  do  not  have  the  resources  to  take  on  such 
a  rol  e  The  M  &S  credi  bi  I  ity  "buck"  appears  to 
stop  atthe(OT)  community,  which  articulated 
the  greatest  caution  aboutM&Susein  acquisi¬ 
tion  of  the  three  communities  represented  at 
the  Workshop  (OSD,  Programs  and  T&E).  It 
was  observed,  howa/er,  that  it  is  too  late  in  a 
prog'am  to  uncover  M&S  problems  or  limita¬ 
tions  in  the  OT  phase  of  an  acquisition  pro¬ 
gram.  Furthermore  the  OT  community  has 
even  more  resource  constraints  than  do  the 
acquisition  programs  themselves.  Thus,  theOT 
community  cannot  be  a  wholly  effective 
"watchdog"  for  M&S  credi  bi  I  ity  in  acquisition. 

Finally,  the  Modeling  And  Simulation  Test 
and  Evaluation  Reform  (MASTER)  concept 
briefed  by  Mr.  O'Bryon  showed  promise  of 
being  able  to  address  some  of  the  technical 
issues  brought  out  at  the  Workshop.  H  owa/er, 
there  was  serious  concern  expressed  by  many 
Workshop  partid  pants  about  MASTER  issues, 
such  as  how  to  fund  the  effort  and  how  to  deal 
with  acquisition  program  control  of  M&S 

Recommendations 

The  briefings,  panel  discussions  and  daily 
summaries  from  the  Reno  M&S  Workshop  all 
seem  to  coalesce  around  the  conclusion  that 
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there  is  no  cause  and  effect  relationship 
between  fol I owi ng  M  &S  and  W&A  pol  icy  and 
technical  guidance  and  having  confidence  in 
M&S credibility  for  acquisition  applications.  It 
seems  dear  that  if  OSD  is  to  reap  the  benefits 
impliat  in  DoD  and  Service  M&S  policies, 
then  the  W&A  "carrot"  for  acquisition 
Progam  Managers  needs  to  be  made  bigger, 
and  the  OSD  "stick"  must  be  more  effective 

In  practical  terms  acquisition  progams 
need  access  to  M&S  technical  support  services 
that  go  beyond  what  the  Service  M&S  Offices 
and  DMSO  can  provide  Acquisition  programs 
need  experienced  assistance  with  meaningful 
implementation  of  M&S  and  W&A  policies 
within  the  constraints  of  their  particular  risk 
and  budget  profiles  On  the  other  hand,  OSD 
needs  to  know  that  M&S  and  W&A  policies 
are  actually  having  an  objective  impact  on  the 
credibility  of  M&S  used  in  acquisition. 

Some  potential  steps  that  OSD  micfit  take 
to  make  the  W&A  "carrot"  bigger  re/olve 
around  providing  acquisition  progams  with 
more  practical  help— 

•  Program-specific  technical  assistance 
geared  toward  meaningful  implementa¬ 
tion  of  W&A  policy  and  technical  guid¬ 
ance  at  a  level  consistent  with  application 
risk  and  progam  resource  constraints 

•  Resources  to  develop  and  archive  W&A 
documentation  for  common  use  M&S 

•Training  in  common,  cost-effective 
approaches  to  W&A  for  acquisition  appli¬ 
cations  of  M&S  so  they  can  do  it  on  their 
own 

On  the  other  hand,  a  sigiificant  step  to 
making  the  M&S  policy  "stick"  big  enough  to 
attract  the  attention  of  acquisition  Progam 
Managers  would  beta  require  acquisition  pro¬ 
grams  to  present  M&S  "Accreditation  Cases" 
for  independent  review  well  before  the  DAB. 
This  would  ensure  that  simulations  that 
adhere  to  M&S  and  W&A  policy  are  in  fact, 
demonstrably  credible  for  their  intended  use 
This  concept  is  consistent  with  Dr.  Gansler's 
memo  of  December  2000  on  "Independent 
Expert  Reviews  of  Software  Intensive  System 
Acquisitions"  It  is  also  consistent  with  some 
international  approaches  to  M&Sa/aluation  in 
an  Acquisition  Reform  environment. 
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On  the  basis  of  the  active  feedback  and  continuing 
interest  in  the  outputs  of  this  workshop  from  many  of 
the  participants,  there  seems  to  be  at  least  prima  fade 
evi  denoe  of  the  val  i  dity  of  these  concerns  at  the  worki  ng 
lewd.  We  recommend  that  OSD  take  short-term  action 
to  establish  thevalidityand  the  magiitude  of  the  issues 
suggested  by  the  Workshop.  It  is  further  recommended 
that  OSD  consider  longterm  options  for  addressing 
both  the  technical  and  the  political  problems  raised  by 
these  issues. 

Two  hi^i-payoff  follow-on  actions  i  n  the  near  term 
are— 

•  Develop  an  implementation  plan  for  MASTER 

•  Develop  data  on  how  acquisition  programs  have 
accredited  M&S  i  n  the  past. 

First,  sincetheJTCG/ASand  theJTCG/ME  co-sponsored 
the  Workshop  under  DOT&E  auspices,  we  recommend 
that  these  two  groups  spearhead  the  development  of  a 
concept  descripti on  and  implementation  plan  for  MAS 
TER.  The  goal  of  this  effort  would  be  to  generate  con¬ 
sensus  in  the  acquisition  community  about  what 
MASTER  should  look  like  in  theory,  and  how  it  might 
work  in  practice  This  would  address  the  concern 
expressed  by  Workshop  parti  a  pants  that  MASTER 
"looks  good  on  paper"  but  that  "the  devil  is  in  the 
details." 

Second,  we  recommend  that  a  short,  focused  study  of 
pre/ious  acquisition-related  M&S  Accreditation  Reports 
and  V&V  efforts  be  conducted.  The  purpose  of  thi  s  effort 
would  be  to  determine  whether  DoD  and  Service  M&S 
policies  were  being  implemented  in  a  manner  that  actu¬ 
ally  provides  objective  e/idence  of  M&S  credibility  for 
acquisition  applications  Itwould  also  attempt  to  deter- 
mi  ne  whether  si  mi  I  ar  standards  of  M  &S  credi  bi  I  ity  were 
bei  ng  appl  i  ed  across  si  mi  I  ar  progams  for  si  mi  I  ar  types 
of  M&S.  This  study  would  address  comments  from  some 
Workshop  parti  a  pants  that,  whilethere  may  be  "loop¬ 
holes"  in  DoD  and  Service  M&S  and  W&A  policies,  they 
are  not  being  widely  used  to  drcumvent  the  need  for 
demonstrated  M&S  credibility  in  acquisition  decision 
making.  This  effort  would  also  extend  the  data  generat¬ 
ed  by  the  1999  DOT&E  M&S  study. 

The  results  of  both  of  these  dTorts  would  be  briefed 
back  to  the  OSD  prinapals  who  partidpated  in  the 
Workshop  and  who  sponsored  the  effort.  The  results 
would  then  be  briefed  to  the  larger  acquisition  commu¬ 
nity  at  the  next  M&S  workshop,  March  4-8,  2002  in 
Reno,  Nevada.  More  information  is  on  this  upcoming 
conference  is  available  at  http://www.nawcwpnsnaw. 
mil/~iasa/. 
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mas  Workshop  2002 

March  4-8  •  Silver  Legacy  •  Reno,  Nevada 

Like  Workshop  2001,  we  anticipate  participation  from  a  wide  spectrum 
oftheM&S  community,  both  Government  and  industry. 


Workshop  2002  will  be  a  logical  sequel  to  the  successful 
2001  Workshop  and  will  address  the  recommendations 
and  condusions  of  the  follow-on  actions  presented  in  the  article 
written  by  Mr.  David  Hall.  The  primary  goal  of  the  workshop  is  to 
enhance  the  way  we  do  business  in  survivability,  vulnerability, 

I  dthal  ity  wi  th  the  empl  oyment  of  M  &S.  Reachi  ng  out  across  DoD 
for  broad  partiapation  in  the  planning  and  execution  of  this 
workshop  a  Workshop  Executive  Advisory  Committee  (WEAC  - 
pronounced  we-ack)  has  been  created.  The  WEAC  is  currently 
defining  the  workshop  spedfictopics  and  issues  in  support  of  the 
primary  goal  with  the  foil  owing  objectives— 

•  Da/el  op  consensus  on  problems*  i  ssues  and  obstades  to  cred¬ 
ible  integration  of  M&S  into  acquisition  programs  and  T&E. 

•  Da/el  op  suggestions  for  better  integration  of  acquisition  pro¬ 
gram  M&S  needs  into  OSD  and  Servicepoliaes,  guidance  and 


initiatives 


•  Identify  potential  solutions,  approaches  and  investment 
strategesfor  M&S  in  acquisition  programs  and  T&E. 


I  n  order  to  achia/e  the  workshop  objectives,  we  have  planned  to 
promote  more  audience  parti  ci  pation  by  creati  ng  worki  ng  groups 
to  define  solutions  to  stated  issues  and  set  aside  more  time  for 
panel  discussions  and  summaries 

The  WEAC  currently  consists  of— 

Simone  Youngblood  (DMSO) 

Mike  Wei senbach  (JTCG/AS) 

Bob  Cook  (USAF) 

Debra  Ridgeway  (USA) 

Frank  Gray  (AFOTEC) 

John  Haug(ATEC) 

Brian  Hall  (OPTEVFOR) 

Jim  Sebolka  (Wash.  I nst.of  Technology— in  support  of  OSD) 
Doug  Fraedrich  (NRL) 

Hans  Mai r  (IDA) 

Tracy  Sheppard  (Univ.  of  Texas  Austin— in  support  of  OSD) 
Kathy  Russel  I  (JASA) 

Ron  Ketcham,  Chairman  (JASA) 


An  informational  brochure  and  Call 
for  Abstracts  was  rel  eased  i  n  O  ctober. 
The  tentative  agenda  isas  follows— 

Monday—  March  4 

Early  Registration 
W&A  Tutorial 
Evening  Mixer 
Tuesday- Th  ursday 

D  ai  I  y  Key  N  ote  Speakers 
Presentations 
Working  Groups 
Panel  Discussions 
Daily  Summaries 

Sponsored  by  the  JTCG/AS  in  coop¬ 
eration  with  DOT&E/LFT  and  coordi¬ 
nated  by  the  Joint  Accreditation 
Support  Activity  (JASA)  and  the 
Workshop  Executive  Advisory 
Committee  (WEAC) . 


For  more  information  contact— 

Ron  Ketcham  760.939.2363 
KetchamRL@navair.navy.mil  or 
Kathy  Russell  760.939.4908 

Russel  I  Ka2@  navai  r. navy,  mi  I . 

Also  keep  checking  the  JASA  Web 
site  for  continual  updates 
http://www.nawcwpns.navy.mi  l/~jasa( 
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National  Summit  on  U.S.  Defense  Policy — 
Acquisition,  Research,  Test  and  Evaluation 

by  M  r.  Tracy  Sheppard 


The  annual  NDIA-sponsored  National 
Test  and  Evaluation  Conference  was 
held  in  Long  Beach,  California  this 
year.  With  the  change  in  Administrations,  it 
was  appropriate  that  this  year's  e/ent  be  a 
National  Summit  on  U.S.  Defense  Policy: 
Acquisition,  Research,  Test  and  Evaluation. 
Panel  sessions  covered  a  range  of  topics  critical 
to  the  state  of  defense  withi  n  the  U  nites  States 
and  its  allies.  The  list  of  speakers  addressing 
the  conference  was  as  diverse  as  the  topics 
themselves 

The  conference  began  with  five  tutorial  ses¬ 
sions  covering  topics  such  as  acquisition 
reform,  defense  policy,  and  the  use  of  non- 
lethal  weapons  in  the  Middle  East.  The  latter 
tutorial  was  presented  by  Colonel  Amir 
Ellenbogen  of  the  Israeli  Defense  Forces  and 
was  a  thoucjit  provoking  and  shocking  exami¬ 
nation  of  the  current  state  of  affairs  in  the 
Middle  East  Needless  to  say,  a  lively  discus¬ 
sion  followed  thistimely  and  important  pres¬ 
entation.  The  tutorials  addressing  acquisition 
and  defense  pol  icy  were  given  by  Government 
and  industry  representatives  alike  and  broucfit 
to  light  both  the  advances  madein  acquisition 
reform  as  wel  I  as  the  fact  that  there  i  s  much  yet 
to  do.  And  with  that  charge  the  conference 
began  in  earnest  the  following  morning  with 
an  address  to  the  conference  attendees  by 
Congressman  Stephen  Horn  (R-CA). 

Congressman  Horn  provided  a  videotaped 
statement  during  which  he  asked  the  attendees 
to  take  seriously  the  conference  objective  of 
providing  a  white  paper  of  recommendations 
to  the  Bush  Administration  addressing  issues 
rancjng  from  National  Missile  Defense  the 
state  of  readiness,  and  international  coopera¬ 
tion  and  theroleoftheprivatesector.  In  addi¬ 
tion  to  Congressman  Horn,  keynote 
presentations  were  given  by  the  Conference 
Chairman,  Mr.  James  O'Bryon,  Deputy 
Director,  Operational  Test  and  Evaluation/Live 
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Fire  Testing,  the  Honorable  Philip  Coyle  former 
Director,  Operational  Test  and  Evaluation,  Dr.  Frank 
Fernandez,  former  Director  of  the  Defense  Advanced 
Research  Projects  Agency,  and  providing  an  industry  per¬ 
spective  Mr.  George  Muellner,  Vice  President  and 
General  Manager,  Phantom  Works,  The  Boeing 
Company. 

Thebusi  ness  of  addressing  the  issues  to  be  addressed 
i  n  the  Whi  te  Paper  began  with  the  keynote  presentati  ons 
and  conti  nued  to  the  end  of  the  e/ent.  Panel  sessions 
were  convened  to  address  each  of  the  conference  topics: 
National  Missile  Defense  national  security,  internation¬ 
al  cooperation  and  the  global  marketplace  the  role  of 
government  laboratories  in  national  defense  commer- 
aal  products  in  national  defense  defense  procurement, 
emerg ng  threats,  and  test,  e/aluation  and  acquisition. 
In  addition  to  these  panel  sessions,  a  Town  Hall  meet¬ 
ing,  a  trademark  of  NDIA'sTest  and  Evaluation  confer¬ 
ences  provided  an  opportunity  for  session  chairs  and 
the  audience  to  discuss  candidly  their  thoughts,  con¬ 
cerns  and  proposals  for  correcting  the  defiaenaes  dis¬ 
cussed  during  the  panel  sessions.  Out  of  this  discussion 
emerged  the  key  points  to  bediscussed  and  addressed  in 
the  conference  White  Paper.  Following  is  a  partial  list  of 
those  points— 

•Attracting  younger  workers  into  the  defense 
industry 

•  I  mprovi  ng  busi  nessTi  nances  of  i  ndustry  versus  con¬ 
ti  nui  ng  cut-throat  competiti  on 

•  Contactor  partiapation  in  operational  testing  for 
learning 

•  Adequacy  of  test  funding 

•  A  business  model  for  Defense 

•  Industry  investments  in  Research  and  De/elopment 

•  Government  review  of  Industry  Research  and 
Development 

•  Impact,  or  lack  thereof,  of  recent  Acquisition 
Reform  changes 

•  Testing  Commercial-of-the-Shelf  products  to  mili¬ 
tary  standards 

The  White  Paper  in  its  entirety,  once  appropriately 
staffed  through  the  Department  leadership,  will  be 
made  available  to  the  conference  attendees  via  the  fol- 
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lowing  Web  sites:  www.dot&osd.mil. www.ndia.ora. 
and  www.csa.utexas.edu. 

I  n  additi  on  to  the  busi  ness  of  the  conference  a  hi  gh- 
I  i  ght  of  the  event  was  a  presentati  on  of  the  annual  N  Dl  A 
awards  for  outstanding  achievement  in  Test  and 
Evaluation.  This  year's  recipients  included— 


al  leadership  and  have  made  si  giifi  cant  contri¬ 
butions  to  the  Defense  community. 

And  thus  ended  another  successful  NDIA 
Test  and  Evaluation  conference  Seniorleaders 
from  Defense  and  industry,  as  well  as  the  hun¬ 
dreds  of  attendees  representing  every  facet  of 


Military 

Civilian 

Contractor 

Army 

Capt  John  Eggert 

Glenn  McPherson 

Charles  Ramsdell 

Navy 

LCDR  Michael  Dodick 

Luis  Cortes 

Robert  Rosado 

Air  Force 

LtCol  David  Nelson 

James  Keith 

Chari  esTri  ska 

DoD 

CDR  Michael  Stanton 

William  Colson 

Dr.  Anil  Joglekar 

The  Walter  W.  H  ol  I  i  s  Award  for  Li  feti  me  Achi  evement 
in  Defense  Test  and  Evaluation  was  presented  to  the 
HonorablePhilipE.  Coyle  former  Director,  Operational 
Test  and  Evaluation.  This  award,  first  presented  to  Mr. 
Walter  Hollis,  Deputy  Under  Secretary  of  the  Army  for 
Operations  Research,  recognizes  those  individuals  who 
throughout  their  careers  have  demonstrated  except ion- 


Mr.  Philip  E.  Coyle,  III  was  presented  the  Walter  W. 
Hollis  Award  for  Lifetime  Achievement  in  Defense  T&E. 


the  defense  community,  came  together,  spoke 
candidly,  and  made  this  event  a  complete  suc¬ 
cess.  Congressman  H  orn's  charge  to  take  this 
task  seriously  was  met.  A  wealth  of  important 
and  timely  information  came  forth  as  a  result 
of  this  conference  While  only  a  fraction  of 
that  information  is  presented  here  a  more 
thoroucfi  re/ia/v  of  the  event  can  be  obtained 
from  a  CD-ROM  containing  the  conference 
proceedings  To  receive  a  copy  of  the  CD, 
please  contact  Ms  Debi  Denney  at  debi_den- 
ney@  i  atutexasedu. 

As  indi rated  throughout  the  conference 
there  is  much  yet  to  do.  A  topic  critical  to  the 
changing  test  and  evaluation  and  defense 
acquisition  process  is  the  role  of  developmen¬ 
tal  testing.  This  will  be  the  theme  of  the  2002 
NDIA  National  Test  and  Evaluation  conference 
to  be  held  February  25  throucfi  March  1,  2002 
in  Savannah,  Georgia.  Pleasejoin  usthere 

Tracy  Sheppard  is  the  Technical  Director  of  the 
Washington  Office  of  the  Center  for  Strategic 
Analysis,  University  of  Texas  (UT)  at  Austin. 
Prior  to  joining  the  research  faculty  of  UT,  Tracy 
served  for  over  15  years  within  DoD,  first  as  a 
Marine  and  then  in  positions  at  Aberdeen  Proving 
Ground  and  within  the  office  of  the  Deputy 
Director  for  OT&E/LFT  in  the  Pentagon.  Tracy 
received  his  AS  and  Bachelor  of  Electrical 
Engineering  degrees  from  the  Johns  Hopkins 
University  in  Baltimore,  Maryland. 


Aircraft  Survivability  •  Fall  2001 


31 


5- 8  —  Monterey,  CA 

Aircraft  Survivability  Symposium  2001 

http://www.ndia.org 

26- 29  — Orlando,  FL 

Interservice/Industry  Training,  Simulation 
and  Education  Conference  (l/ITSEC) 
Contact:  bmcdanial@ndia.org 

27- 29  —  Gaithersburg,  MD 

National  Military  Sensing  Symposium — 
Closing  the  Information  Loop  with  the 
Warfighter 

Contact:  IRIA,  734.994.1200 

27-29  — Nellis  AFB,  NV 

Brawler/ESAMs  User  Group  Meeting 

Contact:  Paul  Jeng,  937.421.2712 

DEC 

6- 8  —  Charlottesville,  VA 

BLUEMAX,  ALARM,  RADGUNS  User 

Group  Meeting 

Contact:  Paul  Jeng,  937.431.2712 


COMMANDER 

NAVAL  AIR  SYSTEMS  COMMAND  (4.1 .8  J) 

47123  BUSE  ROAD 

PATUXENT  RIVER,  MD  20670-1547 


14-17  — Reno,  NV 

40th  AIAA  Aerospace  Sciences 
Meeting  Exhibit 
http://www.aiaa.org 


FEB 

25-28  —  Savannah,  GA 

National  Test  and  Training  Conference 

Contact:  kdouglass@ndia.org 


Information  for  inclusion  in  the  Calendar 
of  Events  may  be  sent  to — 

SURVIAC,  Washington  Satellite  Office 

Attn:  Christina  McNemar 

3190  Fairview  Park  Drive,  9lh  Floor 

Falls  Church,  VA  22042 

PHONE:  703.289.5464 

FAX:  703.289.5467 
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